Upload README.md with huggingface_hub

README.md

@@ -1,361 +1,79 @@
----
-language: th
-tags:
-- thai
-- language-model
-- transformer
-- causal-lm
-- modern-architecture
-- rope
-- swiglu
-- rms-norm
-- gqa
-license: apache-2.0
-datasets:
-- custom
----
-
-# ZombitThai-GPT
-
-A cutting-edge Thai language model with **128K token context support**, featuring modern transformer architecture with advanced optimizations for superior performance and long-document processing.
-
-## 🚀 Key Features
-
-- **🔥 128K Context Length**: Process documents up to 131,072 tokens with dynamic RoPE scaling
-- **🌀 Dynamic RoPE Scaling**: Adaptive positional encoding for sequences of any length
-- **⚡ SwiGLU Activation**: Advanced activation function for improved expressiveness
-- **🎯 RMSNorm**: More stable normalization compared to LayerNorm
-- **🚀 Grouped Query Attention (GQA)**: Efficient attention mechanism with reduced KV cache
-- **🇹🇭 Thai-Optimized Tokenizer**: Advanced Thai syllable-aware tokenization
-- **🔧 Hugging Face Integration**: Full compatibility with transformers library
-- **💾 Memory Efficient**: Optimized for long context processing
-- **🎨 Advanced Sampling**: Multiple strategies for high-quality text generation
-
-## 📊 Model Specifications
-
-| Parameter | Value |
-|-----------|-------|
-| Model Size | 25M+ parameters |
-| Vocabulary Size | 50,000+ tokens |
-| Context Length | **131,072 tokens (128K)** |
-| Layers | 12 |
-| Attention Heads | 12 |
-| Key/Value Heads | 8 (GQA) |
-| Hidden Size | 768 |
-| Feed-Forward Size | 3,072 |
-| RoPE Scaling | Dynamic |
-| Architecture | Modern Transformer |
-
-## 💻 Usage
-
-### Hugging Face Integration (Recommended)
-
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-# Load model and tokenizer from Hugging Face
-model_name = "ZombitX64/ZombitThai-GPT-128K"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-    trust_remote_code=True  # Required for custom architecture
-)
-
-# Generate text with 128K context support
-prompt = "สวัสดีครับ วันนี้อากาศดีจังเลย"
-inputs = tokenizer(prompt, return_tensors="pt")
-
-# Use full 128K context for long documents
-outputs = model.generate(
-    **inputs,
-    max_length=131072,  # Full 128K context
-    temperature=0.8,
-    do_sample=True,
-    top_p=0.9,
-    top_k=50
-)
-generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
-print(generated_text)
-```
-
-### Long Document Processing
-
-```python
-# Process very long documents
-long_document = "..." * 50000  # 50K+ token document
-inputs = tokenizer(long_document, return_tensors="pt", truncation=False)
-
-# Generate with full context awareness
-outputs = model.generate(
-    **inputs,
-    max_length=min(len(inputs['input_ids'][0]) + 1000, 131072),
-    temperature=0.7,
-    do_sample=True
-)
-```
-
-### Memory-Efficient Generation
-
-```python
-# For memory-constrained environments
-with torch.no_grad():
-    outputs = model.generate(
-        **inputs,
-        max_length=16384,  # Reasonable context length
-        temperature=0.8,
-        do_sample=True,
-        use_cache=True  # Enable KV caching
-    )
-```
-
-
-### Local Usage Examples
-
-See `example_usage.py` for custom architecture usage example.
-
-## 🎯 Use Cases
-
-- **📄 Long Document Analysis**: Process entire books, research papers, and lengthy documents
-- **💬 Extended Conversations**: Maintain context across very long chat sessions
-- **📝 Code Generation**: Generate code with full project context awareness
-- **🔬 Research Paper Analysis**: Analyze and summarize complex research documents
-- **📚 Educational Content**: Create comprehensive learning materials
-- **🤖 Conversational AI**: Build sophisticated chatbots with extended memory
-- **✍️ Creative Writing**: Generate long-form content with consistent narrative
-- **🔍 Document Q&A**: Answer questions about extensive documents
-- **📊 Data Analysis**: Process large datasets and reports
-- **🌐 Translation**: Handle long-form translation tasks
-
-## 🔧 Advanced Features
-
-### Sampling Strategies
-
-- **Temperature**: Control randomness (0.1-1.5)
-- **Top-k**: Limit vocabulary to top k tokens
-- **Top-p (Nucleus)**: Limit to tokens with cumulative probability p
-- **Repetition Penalty**: Reduce repetitive text
-
-### Memory Optimization
-
-- **KV Caching**: Efficient autoregressive generation
-- **Gradient Checkpointing**: Memory-efficient training
-- **Flash Attention**: Optimized attention computation
-
-## 📈 Performance
-
-### Context Length Support
-- **Maximum Context**: 131,072 tokens (128K)
-- **Dynamic Scaling**: Automatic RoPE scaling for any sequence length
-- **Memory Scaling**: O(n) for RoPE, O(n²) for attention (optimizable)
-
-### Benchmarks
-- **Perplexity**: < 15 on Thai text corpus
-- **Generation Speed**: ~30-50 tokens/second (batch_size=1)
-- **Memory Usage**: 2-8GB depending on context length
-- **Training Stability**: Improved convergence with modern optimizations
-
-### Long Context Performance
-- **16K tokens**: ~2-3GB memory, fast generation
-- **32K tokens**: ~4-5GB memory, moderate speed
-- **64K tokens**: ~8-10GB memory, slower but functional
-- **128K tokens**: ~16-20GB memory, optimized for batch processing
-
-## 🏗️ Architecture Details
-
-### Core Components
-
-1. **Advanced Thai Tokenizer**: Syllable-aware tokenization with 50K+ vocabulary
-2. **Dynamic Token Embedding**: Maps tokens to 768-dimensional vectors
-3. **Dynamic RoPE**: Rotary position embeddings with automatic scaling for long contexts
-4. **12 Transformer Layers**:
-   - **Grouped Query Attention**: 12 heads (8 KV groups) for efficiency
-   - **SwiGLU Feed-Forward**: 3072 hidden dimensions with gating
-   - **RMSNorm**: Stable layer normalization
-   - **Residual Connections**: Optimized gradient flow
-5. **Language Modeling Head**: Probability distribution over vocabulary
-
-### Long Context Optimizations
-
-- **Dynamic RoPE Scaling**: Automatically adjusts position encodings for sequences up to 128K
-- **Memory-Efficient Attention**: Grouped queries reduce KV cache size by 37.5%
-- **Optimized Feed-Forward**: SwiGLU activation for better parameter efficiency
-- **Stable Normalization**: RMSNorm prevents training instabilities
-
-### Training Details
-
-- **Dataset**: Large-scale Thai text corpus
-- **Training Steps**: 100K+ steps
-- **Batch Size**: 8 (with gradient accumulation)
-- **Learning Rate**: 3e-4 with cosine decay
-- **Mixed Precision**: FP16 training for efficiency
-
-## 🔍 Model Architecture
-
-```
-ZombitThai-GPT-128K Architecture
-├── Advanced Thai Tokenizer (50K+ vocab, syllable-aware)
-├── Token Embedding (50K → 768)
-├── Dynamic RoPE Position Encoding (up to 128K tokens)
-├── 12 × Modern Transformer Block
-│   ├── Grouped Query Attention (12 heads, 8 KV groups)
-│   ├── RMSNorm + Residual Connection
-│   ├── SwiGLU Feed-Forward (768 → 3072 → 768)
-│   └── RMSNorm + Residual Connection
-├── Final RMSNorm
-└── Language Modeling Head (768 → 50K)
-```
-
-### Key Innovations
-
-- **Dynamic RoPE Scaling**: Automatic position encoding adaptation
-- **Grouped Query Attention**: 37.5% reduction in KV cache size
-- **SwiGLU Activation**: Better parameter efficiency than standard FFN
-- **RMSNorm**: Improved training stability
-- **Thai-Specific Tokenization**: Optimized for Thai language patterns
-
-## 📁 Files Structure
-
-```
-ZombitThai-GPT/
-├── config.json                    # Model configuration (custom architecture)
-├── tokenizer.json                 # BPE tokenizer
-├── tokenizer_config.json          # Tokenizer settings
-├── special_tokens_map.json        # Special token mappings
-├── generation_config.json         # Generation parameters
-├── pytorch_model.bin              # PyTorch model weights
-├── modeling.py    # Custom model implementation
-├── __init__.py                    # Package initialization
-├── example_usage.py               # Custom architecture usage example
-├── README.md                      # This file
-└── requirements.txt               # Dependencies
-```
-
-## 📋 Requirements
-
-```txt
-torch >= 2.0.0
-transformers >= 4.21.0
-tokenizers >= 0.13.0
-safetensors >= 0.3.0
-accelerate >= 0.20.0
-```
-
-## 🛠️ Installation
-
-### Quick Start (Hugging Face)
-```bash
-# Install required packages
-pip install transformers torch huggingface_hub
-
-# Load the model
-from transformers import AutoTokenizer, AutoModelForCausalLM
-
-tokenizer = AutoTokenizer.from_pretrained("ZombitX64/ZombitThai-GPT-128K")
-model = AutoModelForCausalLM.from_pretrained("ZombitX64/ZombitThai-GPT-128K", trust_remote_code=True)
-```
-
-### For Long Context Processing
-```bash
-# Install additional packages for optimization
-pip install transformers torch accelerate
-# For Flash Attention support (optional)
-pip install flash-attn
-```
-
-## 🚨 Limitations & Considerations
-
-### Current Limitations
-- **Model Weights**: Currently initialized randomly (not fully trained)
-- **Training Data**: Limited to available Thai text corpus
-- **Domain Knowledge**: May not excel in highly specialized domains
-- **Memory Requirements**: 128K context requires substantial GPU memory (16-32GB+)
-- **Generation Speed**: Longer contexts result in slower generation speeds
-
-### Long Context Considerations
-- **Memory Scaling**: Very long sequences (>64K) may require optimized hardware
-- **Generation Time**: 128K context generation can be 10-20x slower than short contexts
-- **Quality Trade-offs**: Extremely long contexts may impact output coherence
-- **Hardware Requirements**: A100/H100 GPUs recommended for optimal 128K performance
-
-### Language Coverage
-- **Modern Thai**: Optimized for contemporary Thai language
-- **Dialects**: May have limited support for regional dialects
-- **Historical Texts**: Not optimized for ancient Thai or historical documents
-- **Code Mixing**: Limited support for Thai-English code switching
-
-### Recommendations
-- **Short Tasks**: Use 4K-16K context for best speed/quality balance
-- **Long Documents**: Use 32K-64K context for document analysis
-- **Maximum Context**: Reserve 128K for specialized long-document tasks
-- **Memory Optimization**: Consider Flash Attention for better memory efficiency
-
-## 🤝 Contributing
-
-We welcome contributions! Please feel free to:
-
-- Report bugs and issues
-- Suggest improvements
-- Submit pull requests
-- Share your use cases
-
-## 📄 License
-
-This model is released under the Apache 2.0 License. See LICENSE file for details.
-
-## 🎯 128K Context Capabilities
-
-### What You Can Do with 128K Context
-
-1. **📄 Document Analysis**
-   - Process entire research papers (50K-100K tokens)
-   - Analyze complete books and novels
-   - Review legal documents and contracts
-
-2. **💬 Extended Conversations**
-   - Maintain context across multi-hour conversations
-   - Remember user preferences and history
-   - Provide consistent personality across sessions
-
-3. **🔧 Code Generation**
-   - Generate code with full project context
-   - Understand and modify large codebases
-   - Maintain consistency across multiple files
-
-4. **📚 Educational Applications**
-   - Create comprehensive study guides
-   - Analyze lengthy educational materials
-   - Generate detailed explanations
-
-5. **🔬 Research & Analysis**
-   - Process scientific papers and datasets
-   - Analyze complex technical documentation
-   - Generate research summaries
-
-### Performance Tips
-
-- **Batch Processing**: Use smaller batches for very long sequences
-- **Memory Management**: Monitor GPU memory usage
-- **Quality vs Speed**: Balance context length with generation quality
-- **Caching**: Enable KV caching for better performance
-
-## 🙏 Acknowledgments
-
-- **Modern Transformer Research**: Inspired by LLaMA, PaLM, GPT-4, and Claude architectures
-- **Thai NLP Community**: Built for the Thai language processing community
-- **Hugging Face**: For providing excellent model hosting and tools
-- **Open Source**: Standing on the shoulders of countless contributors
-- **Research Community**: For advancing the field of large language models
-
----
-
-**🚀 ZombitThai-GPT-128K - Pushing the boundaries of Thai language AI with extreme context understanding**
-
-## 📞 Contact
-
-For questions or support:
-- Join our Discord community [discord](https://discord.gg/xmVABwsDFU)
-- Email: Jonusnattapong@gmail.com)
-
----
-
-**ZombitThai-GPT - Generated with ❤️ for the Thai NLP community**
+---
+language: th
+license: apache-2.0
+tags:
+- thai
+- language-model
+- transformer
+- long-context
+- 128k-context
+- rope-scaling
+- grouped-query-attention
+---
+
+# ZombitThai-GPT
+
+A modern Thai language model with 128K token context support, built with advanced transformer architectures.
+
+## Model Details
+
+- **Model Type**: Transformer with RoPE and Grouped Query Attention
+- **Context Length**: 131,072 tokens (128K)
+- **Architecture**: Modern GPT with dynamic RoPE scaling
+- **Language**: Thai
+- **License**: Apache 2.0
+
+## Key Features
+
+- 🚀 **Long Context**: Supports up to 128K tokens with dynamic RoPE scaling
+- 🧠 **Advanced Architecture**: Grouped Query Attention, RMSNorm, SwiGLU activation
+- 🇹🇭 **Thai Optimized**: Specialized tokenizer with Thai syllable awareness
+- ⚡ **Memory Efficient**: Optimized for long context processing
+- 🔧 **HF Compatible**: Full Hugging Face transformers integration
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+tokenizer = AutoTokenizer.from_pretrained("JonusNattapong/ZombitThai-GPT-128K")
+model = AutoModelForCausalLM.from_pretrained("JonusNattapong/ZombitThai-GPT-128K")
+
+text = "สวัสดีครับ วันนี้อากาศดีมาก"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model.generate(**inputs, max_length=100)
+result = tokenizer.decode(outputs[0])
+```
+
+## Training Details
+
+- **Architecture**: Custom transformer with modern optimizations
+- **Position Encoding**: RoPE with dynamic scaling for long contexts
+- **Attention**: Grouped Query Attention for efficiency
+- **Normalization**: RMSNorm instead of LayerNorm
+- **Activation**: SwiGLU for better performance
+
+## Performance
+
+- Supports sequences up to 131,072 tokens
+- Efficient memory usage with KV caching
+- Optimized for Thai language processing
+- Compatible with standard HF generation pipelines
+
+## Limitations
+
+- Model weights are randomly initialized (not trained)
+- Requires significant computational resources for long contexts
+- Optimized for Thai text processing
+
+## Citation
+
+```bibtex
+@misc{zombit-thai-gpt,
+  title={ZombitThai-GPT: Modern Thai Language Model with 128K Context},
+  author={AI Assistant},
+  year={2024},
+  publisher={Hugging Face},
+  url={https://huggingface.co/JonusNattapong/ZombitThai-GPT-128K}
+}
+```