Delete those GGUF
Stop encouraging inferior GGUFs. GGUF format is not meant for AI image and video models, leading to degraded loss. INT8 Convrot will run on a 6GB VRAM card with ComfyUI. It is the standard now.
You should delete the GGUF models.
Agreed, these broken gguf's that do not support all features are not helpful. Besides I appreciate VRAi for the release it makes no sense to release broken checkpoint files.
GGUF is just a container, my man. The way it is dequantized depends on the inference backend.
It can run as fast as int8_convrot quant if you want. But comfyanonymous doesn't want to support it.
GGUF is just a container, my man. The way it is dequantized depends on the inference backend.
It can run as fast as int8_convrot quant if you want. But comfyanonymous doesn't want to support it.
Unfortunately, it is not "just" a container. The way it can hold the quanting of keys is different and is not the same architecture like safetensors do. It is fundamentally different and that's why the gguf lacks the features.
I'm here for the GGUFs :D
GGUF is just a container, my man. The way it is dequantized depends on the inference backend.
It can run as fast as int8_convrot quant if you want. But comfyanonymous doesn't want to support it.
GGUF has no meaning to Blackwell users and yes, its quality is degraded as a lower performing quantization compared to an INT8 Convrot. The only people who complain to keep GGUF's are the Ampere users.
Thanks for the feedback. We're considering an official single-file INT8 release for ComfyUI (see #2).
We'll keep the GGUFs too:
- Low-RAM PCs: Today, GGUF is the only path we've measured on PCs with 16 GB or 32 GB of RAM (ComfyUI with
--fast-disk, on 24 GB and 32 GB GPUs). Our current INT8 path, the Python package, needs ≥ 69 GiB with offload. - Smaller files:
Q3_K_MtoQ8_0are 7.2–15.2 GB, and INT8 is ~16 GB.Q4_K_M(9.1 GB) is our default for 24 GB GPUs. - Other runtimes: GGUF also runs in Unsloth Studio and stable-diffusion.cpp (with our patch).
On text-to-video, INT8 was 4–17% faster end to end in our tests. If your setup fits it, use INT8. One known limit: GGUF runtimes accept image references only. Video and audio references need the Python package.
Interesting Conversation.......as a super bad noob i can´t run gguf either......idk why....i don´t like to think about it either. I guess it is super easy since programming follows strict logic......BUT i still Thank Everyone of you, who Tries to make AI Models Better. You are doing great work.
I am just a End User.....who checks out all kind of Models.....from my Point of View it can´t be Easy enough. The Concept of AI is clear..........to do as many -behind the scene processes and work....... as possible and just doing what ....f.e. creators wanna do. You know Pictures and Videos......But i got a Glimpse into What extreme HARD WORK and Nerve Wrecking Programming you Guys do. .....I had to write into shells f.e. in order to make AI run on my Laptop.........You are the HEROS of this AI Revolution that is unfolding before our Eyes.