Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winter time the GPU also doubles as a 300W in-house heater. Not to mention avoiding personal data collection and re-selling.
jgalt212 33 minutes ago [-]
> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands.
This is only true if your local model is already resident in RAM / VRAM.
Havoc 41 minutes ago [-]
Saw some measurements on SBC NPUs (3588) and that did seem to have a decent win on power over CPU...but also a perplexity loss relative to CPU so think this will prove quite hard to reliably quantify in practice.
api 2 hours ago [-]
Unless I misread it, are they saying local GPUs use less energy?
That’s surprising, almost unbelievable, due to batching. Local is usually not batched.
frumiousirc 43 minutes ago [-]
> Unless I misread it, are they saying local GPUs use less energy?
You misread it. From the abstract:
> local accelerators achieve at least 1.4× lower IPW than cloud accelerators running identical models
That's "intelligence per watt". They also have IPJ, per Joule.
So, they find local is 40% "dumber" than cloud for the same power or 40% more power for the same "intelligence".
Tables 13 and 14 summarize their IPW and IPJ metrics.
But, to your actual point, I think the "local is 40% dumber per watt than cloud" message is still an understatement. And maybe this is something I failed to find in the paper but they seem to ignore the "idle baseline" costs and talks about explicitly focusing on the power consumption of just the accelerator under load.
There is a large baseline power consumption just to support the accelerator. CPUs, memory, PS losses, network, fans, general environment cooling. This "cost floor" is different for data centers and a "random local computer" and I think must be in favor of data centers which are designed and built with efficiency in mind.
Idleness should also be considered. My local GPUs at $WORK and home are idle more than they are used. Idle time energy in real world scenarios should be somehow attributed to those brief, punctuated times when LLM functions are actually active on the accelerator. Actual, local LLM usage of a GPU is brief (assuming one user per PC). Even with my heavy usage developing s/w I'd guess I heat up a GPU about one hour per day total, sometimes much less. If that is local then one must pay 23 hours of idleness for that 1 hour of "intelligence". Of course a local PC is used for other things and the idleness penalty must somehow account for that. OTOH, data centers try to maximize utilization so their idle time penalty would be much less, perhaps close to zero, by construction.
stymaar 2 hours ago [-]
Small models are much smaller than frontier models though, which is how they end up consuming less energy despite low batch count. (Though with local models growing strong agentic capabilities, batching becomes a reality with local models as well).
kaziava 2 hours ago [-]
this is the metric i've been waiting for. we run everything local (ollama + neo4j) for compliance reasons, so 'quality per watt' is literally our budget line. one data point from our setup: qwen2.5:3b on an m2 macbook handles nl-to-cypher for simple graph schemas at ~3-5s per answer, and the energy cost is a rounding error compared to shipping the same queries to a frontier api. the hard part was never the model though, it was parsing pdfs locally without a vision model. would love to see parsing/ocr covered in future benchmarks.
iLoveOncall 3 hours ago [-]
> We propose intelligence per watt (IPW), task accuracy per unit of power
Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.
utopiah 3 hours ago [-]
I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.
the8472 1 hours ago [-]
Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy.
Even more so on mobile devices.
frumiousirc 38 minutes ago [-]
They also define and measure an "IPJ" as well as "IPW"
> the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule
than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants
The B200 = "cloud", M4 = "local".
So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
_diyar 3 hours ago [-]
> We propose miles per hour (MPH), distance travelled per unit of time
Stupid metric. It‘s not because you spend more time that you travel farther.
s/
Rendered at 12:16:51 GMT+0000 (UTC) with Wasmer Edge.
This is only true if your local model is already resident in RAM / VRAM.
That’s surprising, almost unbelievable, due to batching. Local is usually not batched.
You misread it. From the abstract:
> local accelerators achieve at least 1.4× lower IPW than cloud accelerators running identical models
That's "intelligence per watt". They also have IPJ, per Joule.
So, they find local is 40% "dumber" than cloud for the same power or 40% more power for the same "intelligence".
Tables 13 and 14 summarize their IPW and IPJ metrics.
But, to your actual point, I think the "local is 40% dumber per watt than cloud" message is still an understatement. And maybe this is something I failed to find in the paper but they seem to ignore the "idle baseline" costs and talks about explicitly focusing on the power consumption of just the accelerator under load.
There is a large baseline power consumption just to support the accelerator. CPUs, memory, PS losses, network, fans, general environment cooling. This "cost floor" is different for data centers and a "random local computer" and I think must be in favor of data centers which are designed and built with efficiency in mind.
Idleness should also be considered. My local GPUs at $WORK and home are idle more than they are used. Idle time energy in real world scenarios should be somehow attributed to those brief, punctuated times when LLM functions are actually active on the accelerator. Actual, local LLM usage of a GPU is brief (assuming one user per PC). Even with my heavy usage developing s/w I'd guess I heat up a GPU about one hour per day total, sometimes much less. If that is local then one must pay 23 hours of idleness for that 1 hour of "intelligence". Of course a local PC is used for other things and the idleness penalty must somehow account for that. OTOH, data centers try to maximize utilization so their idle time penalty would be much less, perhaps close to zero, by construction.
Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.
> the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants
The B200 = "cloud", M4 = "local".
So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
Stupid metric. It‘s not because you spend more time that you travel farther.
s/