File size: 3,127 Bytes
f6f0416 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
base_model: HuggingFaceTB/SmolLM2-360M-Instruct
base_model_relation: finetune
tags:
- code
- coding-agent
- tool-use
- function-calling
- small-language-model
- full-parameter-finetuning
- supervised-fine-tuning
- deterministic-verification
- safetensors
- subroutine:search_query_gen
model-index:
- name: Code Search-Query Generator (SmolLM2 360M)
results:
- task:
type: text-generation
name: Code Search-Query Generator
dataset:
name: Held-out HTTPX and Jinja2 oracle benchmark
type: custom
metrics:
- type: accuracy
value: 0.628
name: Success after one schema-feedback retry
- type: accuracy
value: 1.0
name: First-pass schema validity
---
# Code Search-Query Generator (SmolLM2 360M)
This is a **full-parameter supervised fine-tune** of
[`HuggingFaceTB/SmolLM2-360M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) for one narrow,
schema-bound developer-agent subroutine:
> Write an executable search regex from a plain-words request.
The model is one cell from the
[Parameter Floors for Developer-Agent Subroutines](https://github.com/IshaanAyaan/slm-agents)
experiment. Labels are generated by deterministic oracles over real Python
repositories; no teacher model or human judge labels the data.
## Intended Use
Use this checkpoint inside the repository's verified subroutine harness, which
renders the task-specific prompt, parses strict JSON, permits one localized
schema-feedback retry, applies deterministic guards, and falls back to rules
where appropriate. This is not a general coding assistant or chat model.
## Evaluation
Evaluation uses up to 250 examples from HTTPX and Jinja2, both held out
entirely from training. Decoding is greedy.
| Metric | Result |
|---|---:|
| Success after one schema retry | 62.8% |
| First-pass success | 62.8% |
| First-pass schema validity | 100.0% |
| Base instruct success after retry | 0.0% for the base instruct model |
| Rules-only success | 23.7% |
Experiment verdict for this subroutine: **unsolved (best 0.88 @ qwen2.5-0.5b)**.
## Training
- Training examples: 2000
- Epochs: 3.0
- Learning rate: 2e-05
- Effective batch configuration: 32 per device x
1 gradient accumulation
- Maximum sequence length: 2048
- Seed: 0
- Final training loss: 0.409493
- Reproduction hardware: one NVIDIA A100 80GB PCIe
- Source revision:
[`d0fd7bf`](https://github.com/IshaanAyaan/slm-agents/commit/d0fd7bff420c2f2f0446599ca2b169cc4f03b06a)
The dataset was generated from pinned Flask, Click, and Rich repositories for
training/validation. HTTPX and Jinja2 were reserved for testing.
## Limitations
The checkpoint is specialized to one closed JSON schema and should not be
expected to retain broad instruction-following ability. The experiment mixes
two base-model families across its size sweep. Some subroutines are better
served by deterministic rules; consult the verdict above before deployment.
## License
Apache-2.0, following the base model. Experiment code is MIT licensed.
|