I think this project is a very good way to explore the question:
>What's the difference between a model that predicts intelligent output, and intelligence itself?
It doesn't directly attack the question, but gives a neatly constrained thought experiment which you can see running.
nico 2 hours ago [-]
Very very cool! Impressed that it can be done with only fine tuning
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
simedw 1 hours ago [-]
That's very cool project.
I read this blog post a few days ago about one way to look at memory encoding in models https://alexwang.ai/posts/associative-memory/ that you might find relevant
20 minutes ago [-]
andai 3 hours ago [-]
>I used Qwen2.5-Coder-0.5B as the starting point. I needed a small model to get close to realtime performance. Being a coding model, it had also already seen plenty of CLI usage.
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
—Practice, man... Practice...
treetalker 2 hours ago [-]
> … I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Please elaborate.
simedw 2 hours ago [-]
I haven’t run extensive tests yet, but larger models also get better at memory.
oilkillsbirds 3 hours ago [-]
Very cool. Fun project. I feel like there’s a wide field to be explored once you understand fine tuning good existing models
__mp 3 hours ago [-]
It would be interesting to expose this on the internet. I wonder what the scrapers etc… would do with it.
ohadpr 1 hours ago [-]
Very cool to do this as a Terminal!
I did it for web apps last year, basically a web server that feeds all requests into an LLM.
athropic had an experiment that did exactly this. we need more experimentation in this space.
Lerc 2 hours ago [-]
I wonder how much a small model could do if it had a tool calling API for a filesystem, an embedded busybox, and a single output token for 'emit what busybox would do here'. Then presumably the model parameters could learn everything that diverges from what busybox can handle.
alex7o 1 hours ago [-]
I think the toolcalling should be system prompts then the agent is the userspace
Lerc 1 hours ago [-]
That's a whole other kettle of trout. Maybe still worth a go mind.
On thin that would be useful is an entirely new permission system for files to allow for users and agents acting for users.
psandor 3 hours ago [-]
Reminds me of this project from Microsoft. A Quake II “simulation” model where the whole application was just a model.
Wait, how did it `sleep 3`? I thought LLMs didn't have a concept of time
peder 3 hours ago [-]
It’s still rendering frames constantly, no? So it just figures the next X frames are all doing nothing?
simedw 2 hours ago [-]
Normal only keypresses invoke the next ”frame”. There is a sleep token that the runner could use to indicate that time is moving.
sprobertson 3 hours ago [-]
from the post:
Time is an input too. If nothing is pressed, the model gets a WAIT token with a duration instead of a key, which is how snake keeps moving on its own.
yjftsjthsd-h 3 hours ago [-]
That seems as likely as anything, but that just moves the problem to counting, doesn't it? The model gets the duration, but I was under the impression LLMs struggled with the arithmetic that would make that useful.
skohan 3 hours ago [-]
Idk about Qwen 2.5 Coder that this is based on, but modern models don't struggle with arithmetic. I see Qwen 3.8 do long-hand math in thinking traces (ie. it outputs 17+35+12=64 in the thinking trace) so I would imagine this is something they train into the model specifically. Sometimes it creates and runs a simple python script to check its work.
zahlman 9 minutes ago [-]
It's still strange to me that an ability to perform arithmetic (and not just memorize a giant table of results) could emerge from training on language.
skohan 30 seconds ago [-]
In principle a large enough neural network can provably approximate any continuous function. Since arithmetic can be represented with a concrete grammar, it's not surprising to me that an LLM could contain an abstract representation if trained correctly.
donperignon 2 hours ago [-]
I tried it and is not working remotely as close as the demo gif. It keeps telling me that "test1" folder that I want to create already exists (not true), then letting me do "mkdier test2" (sic) but then the folder is not there when ls.
I mean all its expected, this has no state, follow no rules and is making up things from the current context, but is not as "magical" as I thought when watching the demo.
gitowiec 1 hours ago [-]
BUG: Running in Firefox makes every keypress a slow motion show. It decodes every keypress instead of sending a command with Enter key
odo1242 1 hours ago [-]
No, the model is simulating the entire TUI, not just terminal commands. It also actually is that slow (because it's generating a diff for the entire screen each time you make a keypress).
2 hours ago [-]
prodigycorp 3 hours ago [-]
Wow! This this reminds me of when we tried doing this with gpt 3 and.. it worked!
The tmux thing had me confused for a moment until i realized it was emulating tmux. Super fun. Is the model window width aware?
Where is the point of this project if i have terminal ?
YmFzZTY0ZGFkZHk 3 hours ago [-]
Sometimes it's just fun to do something absurd because you can. This is one of the reasons software is exciting.
mehmetoguzderin 3 hours ago [-]
Terminal world models have many applications, but empirical observation of successful pretraining or fine-tuning can be significant for encapsulating from rather expensive Bash traces, such as ones obtained from systems with scarce accelerators, etc.
good-idea 3 hours ago [-]
because curiosity is the most fun of virtues!
aghuang 3 hours ago [-]
Was thinking the same thing. I just run my usual terminal and get to work.
antonvs 2 hours ago [-]
This will allow you to delete your terminal app and, eventually, all other binaries in your system!
dotancohen 3 hours ago [-]
Because it was there
wartywhoa23 1 hours ago [-]
The point is riding the bandwagon that drags a trailer with a couple of power plants, a small lake of water and a gold reserve in it.
Not unlike that American tractor pulling, but for geeks.
Clouds of smoke, weeeeee!
thorum 3 hours ago [-]
I regretfully suspect this actually is the future of UI. You just need smart local models running fast enough to keep the framerate high, but that’s inevitable in the long term, isn’t it?
skohan 3 hours ago [-]
We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.
winrid 2 hours ago [-]
even my ten year old laptop browses the web fine ^_^
If I had to complain about any site/app it'd be datadog... each tab uses like 1gb in firefox.
skohan 32 minutes ago [-]
I think it's hard to understand how much better your experience could be if all your software was 10x faster and using 0.1x the resources, since you can't test both experiences side by side
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
gr_norm 3 hours ago [-]
Why would you do that instead of having the model do the task you're using the UI for? It could generate UI widgets on-the-fly as needed to take richer inputs.
skohan 2 hours ago [-]
It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
hnlmorg 2 hours ago [-]
Probably for the same reasons most of our desktop applications have become slow Chromium instances consuming multiple GBs of RAM, each:
It’s cheaper to hire developers, and quicker to ship software. The user experience be damned.
skohan 2 hours ago [-]
The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
hnlmorg 23 minutes ago [-]
This demo is already using a local model. Which is why I made the Electron comparison.
antonvs 2 hours ago [-]
I don't really disagree, but... why do we use Javascript for so many UIs? Badly designed web pages with ads can easily use gigabytes of RAM.
Edit: oh, I saw you wrote a related comment after I posted this. So you already know that direct LLMs for UIs are all but inevitable...
latentsea 1 hours ago [-]
What if we're really just living in a world model? How big of a GPU is required to run the universe I wonder?
rvz 33 minutes ago [-]
This makes absolutely no sense what so ever.
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
Rendered at 08:53:22 GMT+0000 (UTC) with Wasmer Edge.
>What's the difference between a model that predicts intelligent output, and intelligence itself?
It doesn't directly attack the question, but gives a neatly constrained thought experiment which you can see running.
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
* web demo: https://nicobrenner.github.io/lossy-git/demo.html
* cli version: https://nicobrenner.github.io/lossy-git/
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
—Practice, man... Practice...
Please elaborate.
I did it for web apps last year, basically a web server that feeds all requests into an LLM.
https://www.ohad.com/blog/voidware/
https://www.youtube.com/watch?v=z3pV6FHvcgM
On thin that would be useful is an entirely new permission system for files to allow for users and agents acting for users.
https://www.microsoft.com/en-us/research/articles/whamm-real...
Time is an input too. If nothing is pressed, the model gets a WAIT token with a duration instead of a key, which is how snake keeps moving on its own.
The tmux thing had me confused for a moment until i realized it was emulating tmux. Super fun. Is the model window width aware?
Not unlike that American tractor pulling, but for geeks.
Clouds of smoke, weeeeee!
If I had to complain about any site/app it'd be datadog... each tab uses like 1gb in firefox.
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
It’s cheaper to hire developers, and quicker to ship software. The user experience be damned.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
Edit: oh, I saw you wrote a related comment after I posted this. So you already know that direct LLMs for UIs are all but inevitable...
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.