From my own test.
It's not faster than the unsloth model.
Disclarer: I'm unsing Vulkan on an AMD GC.
Systemerror7A69 9 hours ago [-]
AMD 7900 XTX with Vulkan here as well, wasn't faster on my test either. Might be much different on Nvidia though.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
noir_lord 8 hours ago [-]
4090 isn't that much higher than the XTX (I also have the XTX), it's 1008GB/s (4090) vs 960GB/s for the XTX's.
The 5090 destroys both at 1792GB/s.
It's not really one thing with the nvidia cards best I can tell it's that they compounded incremental gains from software drivers, card kernels and optimization from been the primary choice (plus first mover advantage).
I didn't buy the XTX for AI purely gaming but it's a capable enough local card for running Qwen et al.
Figs 6 hours ago [-]
4090 vs 5090 performance difference is largely GDDR6 vs GDDR7, I think
wtallis 2 hours ago [-]
It's equal parts memory clock and bus width: RTX 4090 is 384-bit wide at 21 Gb/s and RTX 5090 is 512-bit wide at 28 Gb/s.
DiabloD3 10 hours ago [-]
Surprised its not meaningfully slower.
Vulkan and ROCm paths are missing a few optimized versions of the quants they're using.
GrayShade 2 hours ago [-]
I see around 1 tk/s after a while, so perhaps it is.
Schlagbohrer 8 hours ago [-]
I feel weird that I like your typos, because clearly AI did not write your post. Typos have become downright charming and nostalgic for me.
_ache_ 60 minutes ago [-]
Ahahah, thank you.
Yes, I'm not a native English speaker, and I'm definitely not an AI. ;)
I can't say the same thing because I just don't notice typos (mine or others).j I just assume my English is bad.
Isn't the definition of "being human" is "not to be perfect"?
In French, it kinda is. We say "He/she humain after all" to mean that someone made a mistake.
_ache_ 1 hours ago [-]
Ahahah :'D
Yes... I'm not english native. And definitively not an AI.
syntaxing 6 hours ago [-]
I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.
UncleOxidant 2 hours ago [-]
Vulkan or ROCm backend?
I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.
syntaxing 1 hours ago [-]
Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?
UncleOxidant 1 hours ago [-]
I have not measured pre-fill, but it's said to be around 1000.
It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
syntaxing 5 hours ago [-]
Can you point me towards the model you use, both the main model and the flash model? Curious if I can get ~30 with a higher quant.
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
syntaxing 1 hours ago [-]
Thanks! Have you seen issues with quantizing the kv cache?
Schlagbohrer 8 hours ago [-]
Absolute treasure of a website with these graphs, thank you for sharing this. Huge help for me to find a faster model (smaller quantization) for my VRAM.
npodbielski 5 hours ago [-]
Well I tested it on 7900XTX with the same prompts and their draft model gave me about 30t/s. Their own snippet of code with regular MTP model gave me 60t/s.
Also model with their draft answered incorrectly.
With MTP it answered correctly.
Question was: "Does MikroTik CRS312-4C+8XG-RM have combo ports?". The answer is Yes.
nvme0n1p1 4 hours ago [-]
That's not how you're supposed to use LLMs. You shouldn't expect a tiny little local model to know random facts about every obscure consumer product on earth. That's the job of tool calling. At best a model of this size is just giving you a random guess.
You're basically saying "I tried rolling these dice one time, the green dice rolled a 6 and the blue dice rolled a 1, so green dice are better"
serf 4 hours ago [-]
agreed. a niche knowledge callout is about the worst benchmark one can give a smaller model.
smaller models are attempting to distill the useful methodologies, not the license plate number of an obscure extras car on Magnum PI.
that said I wonder if there is a small 'trivia' model out there. Seems like the kinda thing Google would tackle.
npodbielski 4 hours ago [-]
Which was not he point because I was testing their solution for MPT and it was just funny addition. But of course in internet you always will find some 'well akchually' person straight from the meme.
npodbielski 3 hours ago [-]
When I changed the number of draft tokens to 3 in both, it helped and they Draft is actually performing a bit better:
In the comments it reads like bonsei falls apart on longer running tasks.
electroglyph 6 hours ago [-]
prismml's title is very misleading. in their own paper the model is at 75% of coding scores.
txrx0000 12 hours ago [-]
Not really. The largest IQ4_XS quant here is still worth it because Bonsai doesn't offer larger quants. They could beat it if they made a quaternary variant though, I don't know why they're stopping at ternary.
dingdingdang 3 hours ago [-]
Agree here, as it is ByteShape wins practicality wise if the goal is doing actual work with these quants!
rguiscard 11 hours ago [-]
I wonder the same thing for Bonsai 2. ByteShape offers 5 models from IQ2_XXS-2.56bpw (8.8GB), IQ3_XXS-2.88bpw (9.9GB), IQ3_XS-3.01bpw (10.4GB), IQ3_S-3.23bpw (11.0GB) to IQ4_XS-3.84bpw (13.1GB). Their benchmarks show gradual improvement with size and users can pick one to fit theirs need. Bonsai-2-27B now is about 8.6GB. It might be good to have a quaternary version around 10-11GB to fit a computer with 16-24GB RAM.
Since they’re not using a stable number of bits per token, they use their own naming convention.
noir_lord 7 hours ago [-]
> they use their own naming convention.
Seems like a lot of them do, I only compare them within the same repo because there doesn't seem to be a very standard way of saying all the possible combinations/rearrangements.
iker00 6 hours ago [-]
bpw is the way to compare. huggingface has standard tags that must be used so it forces anyone releasing models to choose a tag that doesn't necessarily equal the actual bpw.
tancop 5 hours ago [-]
It's a bad way to compare. Average bpw ignores the fact that some layers can tolerate more aggressive quant than others.
Rendered at 18:00:19 GMT+0000 (UTC) with Wasmer Edge.
Disclarer: I'm unsing Vulkan on an AMD GC.
I assume the limit for me is memory bandwith, as the 7900 XTX has the same bandwith as the 3090 from what I can gather and I already reached ~60 t/s with Unsloth. Those would fit with the numbers Byteshape has for their cards.
4090 and 5090 have much higher bandwith apparently, so on those cards you can probably get much more out of the kinds of performance improvements they are doing.
The 5090 destroys both at 1792GB/s.
It's not really one thing with the nvidia cards best I can tell it's that they compounded incremental gains from software drivers, card kernels and optimization from been the primary choice (plus first mover advantage).
I didn't buy the XTX for AI purely gaming but it's a capable enough local card for running Qwen et al.
Vulkan and ROCm paths are missing a few optimized versions of the quants they're using.
Yes, I'm not a native English speaker, and I'm definitely not an AI. ;)
I can't say the same thing because I just don't notice typos (mine or others).j I just assume my English is bad.
Isn't the definition of "being human" is "not to be perfect"? In French, it kinda is. We say "He/she humain after all" to mean that someone made a mistake.
Yes... I'm not english native. And definitively not an AI.
I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
DFlash2, Q8_0, --spec-draft-n-max=7: https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
Also model with their draft answered incorrectly. With MTP it answered correctly.
Question was: "Does MikroTik CRS312-4C+8XG-RM have combo ports?". The answer is Yes.
You're basically saying "I tried rolling these dice one time, the green dice rolled a 6 and the blue dice rolled a 1, so green dice are better"
smaller models are attempting to distill the useful methodologies, not the license plate number of an obscure extras car on Magnum PI.
that said I wonder if there is a small 'trivia' model out there. Seems like the kinda thing Google would tackle.
- draft: 67.17
- MTP: 64.18
Why they used those examples? Seems strange.
https://news.ycombinator.com/item?id=49746618
Since they’re not using a stable number of bits per token, they use their own naming convention.
Seems like a lot of them do, I only compare them within the same repo because there doesn't seem to be a very standard way of saying all the possible combinations/rearrangements.