File size: 1,370 Bytes
3d854b9 4cebb7d 3d854b9 4cebb7d 3d854b9 4cebb7d 3d854b9 4cebb7d 3d854b9 4cebb7d 3d854b9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
---
license: apache-2.0
base_model: teknium/OpenHermes-2.5-Mistral-7B
tags:
- fine-tuning
- dpo
- arena-dataset
- peft
- lora
- rlhf
datasets:
- lmarena-ai/arena-human-preference-55k
---
# Mistral-7B DPO Model
This model is a Direct Preference Optimization (DPO) version of [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) using LoRA on the Arena Human Preference dataset.
## Training Details
- **Base Model**: teknium/OpenHermes-2.5-Mistral-7B
- **Dataset**: lmarena-ai/arena-human-preference-55k (1000 samples)
- **Method**: Direct Preference Optimization with LoRA (r=16, alpha=32)
- **Training Steps**: 100
- **Learning Rate**: 5e-5
- **Beta**: 0.1
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load base model
base_model = AutoModelForCausalLM.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
tokenizer = AutoTokenizer.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B")
# Load fine-tuned model
model = PeftModel.from_pretrained(base_model, "gCao/mistral-7b-dpo-arena")
# Generate
prompt = "### Instruction:\nExplain machine learning\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```
|