ask

Next Ask

Next Ask decides which unlabeled example to send to a person next when labels are expensive. It ranks the pool by classifier uncertainty, shortlists the 40 least certain items, then picks items that are both uncertain and far from everything already labeled. A small softmax classifier retrains after each batch.

Space: https://huggingface.co/spaces/thelamapi/Next-Ask — label the digits the selector asks for; the classifier retrains in your browser.

Results (10 seeds, 360 held-out test digits, batches of 10, budget 200)

Strategy Area under learning curve Final accuracy Labels to reach 90% (median)
Uncertainty + diversity (Next Ask) 0.8897 ± 0.0066 0.9583 70 (10/10 seeds)
Uncertainty only 0.8837 ± 0.0135 0.9569 70 (10/10 seeds)
Random 0.8508 ± 0.0115 0.9283 115 (10/10 seeds)

Mean test accuracy at selected label counts:

Labels Random Uncertainty Uncertainty + diversity
20 0.6508 0.6608 0.7036
50 0.8386 0.8650 0.8736
100 0.8864 0.9294 0.9314
150 0.9131 0.9447 0.9492
200 0.9283 0.9569 0.9583

Paired by seed (same initial labels): hybrid − random AULC = +0.0388 ± 0.0113 (hybrid better in 10/10 seeds); hybrid − uncertainty = +0.0060 ± 0.0104 (6/10 seeds).

Learning curves

Method

  • Classifier: multinomial logistic regression on the 64 pixel intensities, full-batch gradient descent from zero weights (steps=150, lr=1.0, l2=0.001, chosen on a 200-example validation split, never on test).
  • Uncertainty: margin between the two most probable classes.
  • Diversity: farthest-first selection on the uncertain shortlist, score = (1 − margin) × distance to the nearest labeled or already chosen example.
  • The browser runs a line-by-line JavaScript port. Node parity on 4 fixed cases: max abs difference 1.1102230246251565e-16.

Usage

from inference import load, predict, next_queries
spec = load()                       # model.json, NumPy only
probs = predict(spec, pixels)       # pixels: (N, 64) ints 0..16
picks = next_queries(spec, pool_pixels, labeled_idx, labels, strategy="hybrid", batch=5)

python inference.py --selftest reloads the model and checks stored outputs.

Data

Optical Recognition of Handwritten Digits (8x8) — UCI ML Repository id 80 via sklearn.datasets.load_digits, license CC BY 4.0. Split: 1237 pool / 200 validation / 360 test, stratified, seed 73.

Limitations

  • 8×8 digits are an easy, balanced benchmark; gains from active learning are usually smaller on noisy, imbalanced real data.
  • The simulation assumes a perfect annotator (dataset labels). The Space lets people label, but no human study was run.
  • Final model in model.json is the seed-0 hybrid run at the full budget, kept for demonstration (17 of 360 test digits wrong).
  • Hardware for the recorded run: NVIDIA L4. Profile: full.

Files: model.json (weights), ask_core.py (selectors and training), inference.py, nlrt.py, config.json, evaluation.json (all numbers, seeds, failures, tests).

Downloads last month
25
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Space using thelamapi/Next-Ask 1

Collection including thelamapi/Next-Ask