ViveKi-0.1-20B

A general-purpose open-weight decision model that can natively process text, images and videos.

Built on a ~27B-parameter base model, but named for what actually runs: this model's forward pass exits early at an internal readout layer, so roughly 20B of those parameters ever execute on a real request. See Requirements below.

Demos

Live video monitoring, updating in real time

A monitoring console asks the model a running question against live footage — cheap enough per decision that it re-evaluates several times a second, so the clip below is every decision the model actually made as the footage played, not one end-of-clip verdict. The same question gets asked across a real soccer clip and two real violence-detection clips, back to back: watch it sit at NO through the build-up, flip to YES the instant the ball crosses the line, then separately fire YES on a real fight clip and correctly stay at NO on a real non-fight clip — same question, same model, no per-clip tuning. See demo above.

Quickstart

pip install "transformers>=5.8.0" torch accelerate pillow torchvision flash-linear-attention

Using the pipeline

from transformers import pipeline

decide = pipeline("viveki-decision", model="not-ekalabya/viveki-0.1-20B", trust_remote_code=True,
                   torch_dtype="bfloat16", device_map="auto")

result = decide({
    "image": open("frame.jpg", "rb").read(),   # bytes, base64 str, or a PIL.Image
    "questions": [
        {"instruction": "What object is in the frame?", "options": ["phone", "book", "cup"]},
        {"instruction": "Is a person visible?", "options": ["yes", "no"]},
    ],
})
print(result)  # a single request -> a flat list of per-question results, as below

[
  {
    "instruction": "What object is in the frame?",
    "scores": {
      "phone": {"score": 0.0421, "probability": 0.91},
      "book":  {"score": -0.0117, "probability": 0.06},
      "cup":   {"score": -0.0233, "probability": 0.03}
    }
  },
  {
    "instruction": "Is a person visible?",
    "scores": {
      "yes": {"score": -0.0052, "probability": 0.41},
      "no":  {"score": 0.0019,  "probability": 0.59}
    }
  }
]

score is the model's raw, unnormalized per-option readout — returned by the model itself with no post-processing. probability is a softmax over those scores, computed by the pipeline after the model call, not by the model. Call the model/processor directly (below) if only the score is required, with no softmax at all.

Using AutoModel / AutoProcessor directly

from transformers import AutoModel, AutoProcessor

repo = "not-ekalabya/viveki-0.1-20B"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, torch_dtype="bfloat16").to("cuda")

inputs = processor({
    "video": [open(f"frame_{i}.jpg", "rb").read() for i in range(8)],
    "text": "Security camera feed, loading dock entrance.",
    "questions": [
        {"instruction": "Does the video show a vehicle entering?", "options": ["yes", "no"]},
    ],
})
outputs = model(**inputs)
print(outputs.scores)         # outputs.scores[0][0] -- raw scores only, no softmax applied

A single, non-batched request still comes back with the same nested [item][question] shape as a real batch (see Batching below) — just with one item.

Multiple images + video + text together

images (a list), video (a list of frames), and text can all be given in the same request — they're combined into one shared context that every question in that request is scored against, in one forward pass.

inputs = processor({
    "images": [open("reference_a.jpg", "rb").read(), open("reference_b.jpg", "rb").read()],
    "video": [open(f"frame_{i}.jpg", "rb").read() for i in range(8)],
    "text": "The two reference images show the item as shipped; the video is the unboxing.",
    "questions": [{"instruction": "Does the video show the same item as the references?",
                    "options": ["yes", "no"]}],
})
outputs = model(**inputs)

image (singular) remains as a convenience alias for a single-item images list.

Video inference

video is a list of frames (bytes, base64 strings, or PIL.Images), already sampled by the caller — this model does not decode video files itself. Frames are read in the order given and labelled by index, not by wall-clock time.

inputs = processor({
    "video": frames,  # list[bytes | str | PIL.Image]
    "questions": [{"instruction": "What is happening?", "options": ["loading", "idle", "unloading"]}],
})
outputs = model(**inputs)

Multiple questions in one call

Every question in questions is scored against the same shared images/video/text content, in one forward pass — asking 5 questions costs one pass, not five.

inputs = processor({
    "image": frame_bytes,
    "questions": [
        {"instruction": "Primary object?", "options": ["phone", "book", "cup", "none"]},
        {"instruction": "Lighting?", "options": ["bright", "dim"]},
        {"instruction": "Indoors or outdoors?", "options": ["indoors", "outdoors"]},
    ],
})
outputs = model(**inputs)

Text-only decisions

images/image and video are all optional — a request with just text works the same way.

inputs = processor({
    "text": "Customer wrote: 'this broke after one day, I want a refund.'",
    "questions": [{"instruction": "What is the customer asking for?",
                    "options": ["refund", "replacement", "complaint only", "praise"]}],
})
outputs = model(**inputs)

Batching multiple requests

Pass a list of requests instead of one. Each keeps its own content and its own questions -- they don't share context with each other. Requests are grouped internally by which internal profile their questions need (a small, fixed set of internal readout profiles, chosen per question by its option count, not by modality) and each group gets padded together into one forward pass -- so a batch of same-shaped requests (e.g. all few-option, or all many-option) costs one forward pass total; a batch mixing both shapes costs one pass per shape present, still far fewer than one pass per request. This grouping is invisible to the caller -- results always come back in the same order the requests were given, regardless of how they were grouped internally.

requests = [
    {"image": img_bytes_1, "questions": [{"instruction": "Primary object?", "options": ["phone", "book"]}]},
    {"image": img_bytes_2, "questions": [{"instruction": "Primary object?", "options": ["phone", "book"]}]},
    {"text": "Looks fine to me.", "questions": [{"instruction": "Sentiment?", "options": ["positive", "negative"]}]},
]
inputs = processor(requests)
outputs = model(**inputs)
print(outputs.scores[1])   # second request's per-question scores

# equivalently, via the pipeline:
results = decide(requests)   # returns a list, one entry per request, same order as the input

Request schema

{
  "images": ["<img 0>", "<img 1>", "..."],        // optional, standalone images
  "image":  "<bytes | base64 str | PIL.Image>",   // optional alias for images=[img]
  "video":  ["<frame 0>", "<frame 1>", "..."],    // optional, ordered video frames
  "text":   "optional free-text context",
  "questions": [
    {
      "instruction": "optional, defaults to a generic classification prompt",
      "options": ["<class A>", "<class B>", "..."]   // required, no fixed upper bound
    }
  ]
}

images, video, and text can be combined freely in one request — any subset of them becomes one shared context that every question in that request is scored against. At least one of images/image/video/text must be given, and questions must be non-empty.

Batching: pass a list of request dicts instead of one to AutoProcessor.__call__ (or to the "viveki-decision" pipeline) to score several independent requests in a single forward pass. Requests in a batch don't share context with each other — only images/ video/text within one request are shared across that request's own questions.

There is no fixed cap on options per question, images/frames per request, or questions per request — every option, image, frame, and question becomes real tokens in the shared forward pass, so the only real limit is the base model's own context window (262,144 tokens). Option labels aren't limited to A-Z either: past 26 options a question's labels continue AA, AB, AC, ... (spreadsheet-column style), not wrapping back to A.

Hyperparameters

Name	Where	Default	Meaning
softmax_temperature	pipeline call, e.g. decide(request, softmax_temperature=0.05)	repo default	Temperature of the softmax the pipeline applies to raw scores. Lower = sharper, more confident probabilities. Does not affect score, only probability.

Output

The model (ViveKiForDecision.forward) returns a ViveKiOutput with:

• scores: list[list[list[float]]] — scores[i][j][k] is request item i's question j's option k's raw score. A single, non-batched request still has this shape, with a leading dimension of 1.
• option_labels: list[list[list[str]]] — the option strings, same shape, for mapping scores back to names.
• instructions: list[list[str]] — the instruction text per request item per question, passed through unchanged.

The pipeline ("viveki-decision") wraps this into the JSON shown above (a list per request item, of per-question results), adding a probability field per option via a softmax over score — nothing else is changed from what the model itself returned. A single (non-list) request to the pipeline is unwrapped back to a flat per-question list, matching the Quickstart example above.

Requirements

• transformers>=5.8.0, torch>=2.12.0, accelerate
• flash-linear-attention — required, not optional. This model's base has linear-attention layers that fall back to a much slower reference path without it.
• The full ~27B-parameter base model loads into memory (its weights aren't split and re-hosted at a smaller size) even though the forward pass only executes roughly 20B of those parameters per request — the early exit saves compute, not VRAM. A GPU with enough memory for a 27B-parameter model in bfloat16 (e.g. an 80GB-class card) is still needed. device_map="auto" will shard across multiple smaller GPUs if needed.
• trust_remote_code=True on every from_pretrained/pipeline call — this repo ships custom model, processor, and pipeline code.

License

Apache 2.0 for the code in this repository. Using this model also means loading and running the base model it wraps (Qwen/Qwen3.8-27B) — see that repository for its own license terms.