It would be great to see Apollyon on Qwen 3.5 or 3.8, especially if it’s based on a version with distilled reasoning (for example, from TeichAI). That would be absolutely amazing!
i’ll keep that in mind when creating my next version. The problem has been that the reasoning models that i would use to distill the reasoning from would probably censor the thinking steps as the conversations i use to train my models are very explicit.
I can feel the concern. I did the following originally as kinda needle test. But I suspect the model did not just missed the needle. I attached Richard Morgan's full novel Broken Angel (>100K tokens, @256k ctx) to the conversation and asked Gemma 4 12B,26B:
How many Wedge soldiers did Kovacs kill and who are they? - Both: none !
What happened to Tony Loemanako ? - Both: the noncom appeared in CH2 but there's no mention of what happened to him at last.
What happened in CH39? - (make up stories, severe hallucination, completely off. Actually the novel said K rushed his windpipe at the end of that chapter.) Qwen 4B: (Basically it's unable to understand the novel)
The provided text did not say how many! Qwen 9B: (minor hallucination on the detailed plot in reasoning)
2 named: Tony Loemanako, Carrera and several others. Qwen 30B 3AB:
9 named: ... and several others (in a nice table with chapter# and details)
I tried several Gemma quants Q4/Q6, unsloth and different uncensored versions of 12B and 26B. I'm surprised even at 26B, none of them can answer the question correctly.. I mean not even close. They all exhibited erratic thinking patterns. Instead of recalling the details, it summarized each chapter into a super short one-liner and concluded no. Then start the but wait loop on different chapters in random orders, however, Tony in CH39 was never mentioned.
