tkatz123/qwen2.5-3b-job-extraction-gguf

Qwen2.5-3B Job-Extraction — Q4_K_M GGUF

A Qwen2.5-3B-Instruct model fine-tuned (QLoRA) for structured extraction from job postings: raw posting → JSON of {required_skills, tech_stack, seniority, avg_comp_range}. This repo holds the Q4_K_M GGUF quantization (~1.8 GB) for CPU serving via llama.cpp / llama-cpp-python.

Part of a portfolio MLOps project (fine-tune → containerize → deploy on AWS).

Related artifacts

• LoRA adapter: tkatz123/qwen2.5-3b-job-extraction
• Merged fp16 model: tkatz123/qwen2.5-3b-job-extraction-merged
• Quantized GGUF (this repo): built from the merged model with llama.cpp convert_hf_to_gguf.py + llama-quantize (Q4_K_M).

Results — fine-tuned vs. same base model, zero-shot

Field-level accuracy on a held-out test set (29 postings), scored under identical greedy decoding:

Field	Base (zero-shot)	Fine-tuned	Δ
seniority	0.72	0.76	+0.03
comp (±10%)	0.86	0.97	+0.10
skills (set-F1)	0.17	0.27	+0.10
tech_stack (set-F1)	0.08	0.68	+0.60
valid-JSON rate	1.00	1.00	—

The largest gain is tech_stack (0.08 → 0.68): fine-tuning taught the model to cleanly separate named technologies (Python, PyTorch, AWS, Docker) from general competencies.

Usage (llama-cpp-python)

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="tkatz123/qwen2.5-3b-job-extraction-gguf",
    filename="qwen2.5-3b-job-extraction-Q4_K_M.gguf",
    n_ctx=4096,
)

resp = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": EXTRACTION_PROMPT},   # see project repo
        {"role": "user", "content": job_description},
    ],
    max_tokens=1024,
    temperature=0.0,   # greedy, matches evaluation
)
print(resp["choices"][0]["message"]["content"])

Output is a JSON object: {"required_skills": [...], "tech_stack": [...], "seniority": "...", "avg_comp_range": <int|null>}.

Serving notes

Served as a FastAPI + Docker microservice on an AWS EC2 Graviton instance. The Q4_K_M quant needs ~4 GB RAM to serve comfortably (on a 2 GB box the weights don't stay resident and inference becomes disk-bound).