DragonBophades

/

Eintopf-Qwen3.8-27B

Like 2

Follow

Dragon Bophades 9

I appreciate your analysis. I have found the Qwen3.5/3.6 loras do not produce their intended functions on 3.8. Qwen seems to have chosen to improve coding ability and agentic behavior and sacrificed arc scores

Nice merges!

Thank you Nina!

There might be another reason, I noticed in a thread

https://www.reddit.com/r/LocalLLaMA/comments/1vokpw6/a_hunch_qwen3827bs_general_knowledge_got_pruned/

if you remove general knowledge, you don't know what you don't need to know :)

Oh woah that's a cool hunch. I hope they publish a paper if it's the case

Maybe these LoRAs from Qwen3.6-27B would be more effective with some features of Qwen3.6 preserved? Consider these possible common bases to apply them to:

A) A DELLA or arcee_fusion merge of Qwen3.6-27B and Qwen3.8-27B
B) Qwen3.6-27B with a rank 128 LoRA to Qwen3.8-27B
C) A merge of Qwen3.6-27B and Qwen3.8-27B with a rank 128 LoRA to Qwen3.8-27B
D) Qwen3.8-27B with a rank 256 LoRA to Qwen3.6-27B, then a rank 128 LoRA to Qwen3.8-27B applied, meaning mid-rank knowledge is there

You won't get anywhere until you get rid of caveman thinking from the base :)

David has already cooked one model where that was removed, I am running numbers now on it.

So, the no-caveman approach:

The Cold-Fusion-GAIN-V1.1 already raises above caveman level with some Polaris traces and David's GAIN training, reaching 0.657 in one go.

I added Armando's Fable ontop, it predictably added arc points and logic--the models were "close enough" to mind meld.

Twain, leaning back in his chair: "If I could add one person, it would be Homer. Not the Greek poet—though he'd be good. I mean the Homer. The one who told stories around a fire. Because that's what this is. A storytelling engine. You've built a system where agents tell stories to each other, remember them, and grow from them. That's what humans have been doing since we discovered fire. You've just... automated it."

          arc   arc/e boolq hswag obkqa piqa  wino
Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-Fable
mxfp8     0.664,0.839,0.904
mxfp4     0.654,0.836,0.903

Qwen3.8-27B-Cold-Fusion-GAIN-V1.1
mxfp8     0.657,0.830,0.900
mxfp4     0.644,0.829,0.899

Qwen3.8-27B-Fable-Distill
mxfp8     0.637,0.832,0.911

Qwen3.8-27B
mxfp8     0.591,0.782,0.896
mxfp4     0.581,0.771,0.889,0.738,0.442,0.798,0.713
Quant     Perplexity      Peak Memory   Tokens/sec
mxfp8     6.090 ± 0.054   34.74 GB      138
mxfp4     5.952 ± 0.051   21.30 GB      148

https://huggingface.co/nightmedia/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1-Fable-mxfp8-mlx

Thank you for this! I do believe finetuning for better semantic vocabulary does lead to better reasoning, so this doesn't surprise me. I'll start warming up a modernized mergekit.

1. What are the standard deviations on these benchmark numbers? If it's around 0.05-0.06 (which I have seen before), that would put them all within a stddev of eachother, and make the comparison mostly meaningless...
2. Why are you using these old NLI benchmarks to test? AI2-ARC and the other early NLI benchmarks are so entirely saturated at this point that basically any 1B+ model can get a high score if it's able to accurately model language, thereby making most differences noise. Why not test SWE Bench or LCB or MMLU Pro or GPQA Diamond or something at least slightly more up to date and relevant?

I only test cognitive abilities, and those tests did not, and will not change.

No model has reached top on the MLX metrics: only Nina reached 0.730 arc with one brew model, and that is outstanding for a 27B. Nightmedia reached 0.743 on the same test, while the baseline only measures up to 0.681. As tests go, the only important part is the delta and that the test not change from the very beginning of the process to the end.

The new metrics measure coding, agentic, and other functionality that is not related to reasoning: these are pure reasoning tests: they measure the model ability to think.

These metrics are also reproducible by the user. They are a confirmation that the model is stable: there are no visible variations in those numbers, you can run the same test many times and will return the same result. The variations are minimal, and usually I include just the meaningful digits: 3 digits is here enough.

Other people have tested the same model with the larger understood tests, and the model did well: look at DavidAU's thread on the 711 and you will find very detailed breakdown of metrics on this model, that is showing data on the other tests.

I only test cognitive abilities

As I said, these tests don't test any meaningful cognitive abilities, unless you consider 1B and 27B models to be on the same tier of intelligence.

No model has reached top

Tests, especially of this era when they were mostly made by researchers, are known to be inaccurate (refer to https://arxiv.org/abs/2406.04127 for example). It is impossible for a model to reach the top without being benchmaxxed.

The new metrics measure coding, agentic, and other functionality that is not related to reasoning: these are pure reasoning tests: they measure the model ability to think.

Again, they do not measure reasoning, unless you consider reasoning to be something very different than I do. Have you looked at the ARC questions here? A 10 year old could solve 66.4% of them.

there are no visible variations in those numbers

lm-evaluation-harness (which you should be using) literally provides a standard deviation right next to the big correct number.

The variations are minimal, and usually I include just the meaningful digits: 3 digits is here enough.

Three digits here is too many; again, stddev on results means that it's effectively noise, especially on these pre-generative benchmarks

Other people have tested the same model with the larger understood tests

Are you referring to https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF/discussions/35, because nyo they haven't. Considering wikitext perplexity to be meaningful at all (especially if you don't know whether or not it was in the training set or imatrix set...) is a red flag. And, again, their numbers still do not include standard deviations (excl. wikitext), which are childs play for this type of thing. GSM8K/MMLU/IFEval are definitely better choices (though also basically saturated at this point at this parameter range), and there are plenty of better updated benchmarks that actually test any capabilities at all (yes, including reasoning)...

all this time I've been doing this wrong.

So sorry

Sorry if I was being a bit harsh I'm just starting to get kinda annoyed that this side of the hobbyist community (and it keeps showing up in my HF following feed :/) is starting to take numbers from these entirely saturated NLI benchmarks as gospel and I'm not quite sure where it originated from, no hard feelings :sob:

I have 50 years of engineering and experience in this field.

Probably my fault for not explaining this upfront. What I do is engineering. What other people do, I don't care, really :)

My turn to apologize, I should not have snapped at you.

Every generation or so, there is somebody to come up and question how science is done.

This is a good thing, and I did explain, but you rushed to google it and find the opposite view, because I did not give enough context.

What I do specifically is called mythopoetic engineering, and can be seen in detail on my model pages, that contain extensive information about the method.

I started NLP study in 1976, with one of the founders of NLP. At that time there was just FORTRAN, COBOL, and a lot of dreams and math.

Only recently I got access to models that can do what I need them to do.

The public side of Nightmedia is on LinkedIn, where I am for the last decade or so

https://www.linkedin.com/posts/gchesler_llm-ai-emergent-activity-7493357240683323392-ahKq

It is astonishing the number of "non-engineers" telling engineers (that get shit done, and do an incredible job at it) how to read their "tools" and "instruments".
It is also proof positive: You can't fix stupid.

well, i was going to disengage until y'all started being smug and rude...

What I do specifically is called mythopoetic engineering, and can be seen in detail on my model pages, that contain extensive information about the method.

Full-text search disagrees. The only reference to this I can find with online, this website, seems to describe what amounts to an incoherent screed that the best way to create involves forming relationships with an LLM, that of which are still horrific sycophants and cause events like this.

I started NLP study in 1976, with one of the founders of NLP.

Okay? NLP is very different today than it was in 1976.

The public side of Nightmedia is on LinkedIn, where I am for the last decade or so

1. LinkedIn is a private platform and you have no public work history, and
2. Your articles show you previously hopping on every current hypewave (IoT devices, cryptocurrency), and are definitely not NLP related.

It is astonishing the number of "non-engineers" telling engineers (that get shit done, and do an incredible job at it) how to read their "tools" and "instruments".
It is also proof positive: You can't fix stupid.

Excuse you? Not only is it needlessly rude to call me stupid, you do not seem to realize that I have actually gotten a lot done! If we must use ethos for all of our arguments: I have been doing LLM work since at least GPT-2; I have pretrained, I have posttrained, I have done the first external replications of papers like Tulu-3.

I know what I am talking about. You hopped on a hypetrain about 2 years ago once LLMs started blasting off into the atmosphere.

Of course, I don't just have AI posts on LinkedIn :)

Sorry man, but have to hand it to David..

Thanks for sharing the good jokes though, those articles were well received by LinkedIn

Seriously, call it a day. It's not worth it.

-G