Article
Does Your LLM Know *When It's About to Be Wrong*?
ginigen-ai
โข โข 17ginigen-ai PRO
ginigen-ai
AI & ML interests
None yet
Recent Activity
updated a Space about 1 hour ago
ginigen-ai/Metacognition-Leaderboard-Space repliedto SeaWolf-AI's post about 2 hours ago
๐ฏ Chitos โ The Security Scanner That Actually Proves It
Most security scanners hand you a suspect list and walk away. That gap between detection and proof is where attackers live โ and it's exactly the gap that Chitos was built to close.
Chitos is the successor to Mythos, a static analyzer built for quick code health checks. Mythos was good at pattern matching โ spotting dangerous sinks, mapping CWEs, producing readable reports. But static analysis has a structural ceiling. A rule that sees eval(user_input) can tell you that looks dangerous. It cannot tell you whether the input is reachable, whether sanitization three layers up covers this path, or whether there's a live exploit chain for your exact framework version. Chitos was built to answer those questions.
๐ Phase 1 applies 50 language-agnostic rules across Python, JavaScript, Go, Java, C/C++, Rust, PHP, YAML and more โ covering injection sinks, deserialization gadgets, credential leakage, broken crypto, and prototype pollution. Every candidate is re-verified before reaching the report. Findings that can't be substantiated are excluded, not handed to you as noise.
๐ฌ Phase 2 dispatches an autonomous web-search agent to hunt live CVE databases, exploit advisories, and public PoC repositories. It formulates hypotheses, verifies them, and synthesizes a structured threat narrative. This phase needs a user-supplied Claude API key โ Phases 1 and 3 run entirely free.
๐ฏ Phase 3 is where Chitos diverges from everything else. Against targets you own or are authorized to test, it fires real payloads โ XSS, SQLi, path traversal, command injection โ mutates on block, captures hard evidence, and connects every proven finding into a kill-chain showing which vulnerabilities to remediate first.
No installation. No account. No code sent to third-party APIs.
Article: https://huggingface.co/blog/FINAL-Bench/chitos
Try it now ๐ https://chitos.vidraft.net
repliedto their post about 15 hours ago
๐ง Does your LLM know when it's about to be wrong?
Most leaderboards measure accuracy. We measure metacognition โ whether a model catches its own errors. Benchmark + leaderboard + adapters, all open. ๐
The surprise: even a K-AI #1 model (JGOS-31B-Citizen) is the strongest on multiple-choice traps (trap_rate 0.005 โ ~2 misses in 400) yet blind to its own free-form mistakes (self-confidence AUROC = 0.5, pure random). A tiny base-frozen adapter recovers that signal.
Two independent axes (never compared across a row): โ trap_rate โ does it fall for tempting trap options? (lower = stronger) โก adapter gain ฮ โ how much a lightweight adapter catches errors the model itself misses. (higher = more adapter value)
What's open: ๐ 300+100 trap problems (each with a hidden trap + TICOS type) ๐ 24-model leaderboard ๐งฉ 11 per-model adapters โ adapters, NOT fine-tunes (base stays frozen; the adapter just reads the hidden state โ P(wrong))
Submit any HF model โ auto-scored daily at 09:00 KST and added to the board.
๐ Leaderboard โ https://huggingface.co/spaces/ginigen-ai/Metacognition-Leaderboard-Space
๐ Benchmark โ https://huggingface.co/datasets/ginigen-ai/Metacognition-Bench
๐งฉ Adapters โ https://huggingface.co/collections/FINAL-Bench/metacognition-adapters-6a42c032e6beb803dd032961
๐ Article โ https://huggingface.co/blog/ginigen-ai/metacognition
Benchmark by ginigen-ai ยท Adapters by FINAL-Bench (Darwin/Chimera platform + AETHER metacognition tech).Organizations
None yet
published an article 2 days ago