CloudGoat/Qwen3-Hydra-12B

Qwen3-Hydra

Qwen3-Hydra is a Mixture of Experts (MoE) created from the following models using Updated LazyMergekit by CloudGoat:

• Qwen/Qwen3-4B-Instruct-2507
• zenlm/zen-agent-4b
• InternScience/Agents-K1
• AliesTaha/fable-traces

🧩 Configuration

base_model: Qwen/Qwen3-4B-Instruct-2507
gate_mode: hidden
dtype: bfloat16
experts:
  - source_model: Qwen/Qwen3-4B-Instruct-2507
    positive_prompts:
      - "general instruction following, logical reasoning, and helpful response"
      - "explain the concept clearly and answer the user question"
  - source_model: zenlm/zen-agent-4b
    positive_prompts:
      - "function calling, API usage, and external tool integration"
      - "execute the agent action using available tools and functions"
  - source_model: InternScience/Agents-K1
    positive_prompts:
      - "scientific reasoning, knowledge graph retrieval, and paper analysis"
      - "extract structured scientific knowledge and perform multi-hop research reasoning"
  - source_model: AliesTaha/fable-traces
    positive_prompts:
      - "fable method execution, step-by-step verification, and trace planning"
      - "follow constrained agentic workflow and report outcome with caveats"

💻 Usage

!pip install -qU transformers bitsandbytes accelerate

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "CloudGoat/Qwen3-Hydra"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Explain what a Mixture of Experts is."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))