tokenizer: expose .vocab property for fast-tokenizer-style callers

by kashif HF Staff - opened 7 days ago

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

-0

kashif

Hugging Face Biology Research org 7 days ago

Adds a vocab property mirroring get_vocab() so downstream tools that expect the fast-tokenizer interface (e.g. llama.cpp's convert_hf_to_gguf.py which does tokenizer.vocab) work without a fallback. No behavior change.

tokenizer: expose .vocab property for fast-tokenizer-style callers4c899c4a

kashif changed pull request status to merged 7 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment