From what I'm experiencing, we have recursive self improvement that can find local maxima, I'm not seeing really strong evidence of unguided RSI that finds the true optimal.
Sharlin 8 hours ago [-]
Doesn’t matter if there’s a rechable local maximum that happens to be comfortably beyond human level. Indeed it would be very surprising if a thing capable of RSI just happened to get stuck at human level or slightly above or below, even though its constraints are entirely different from those evolution had to work with when it created us!
FloorEgg 3 hours ago [-]
What if you are underestimating some qualities of our intelligence and how efficient they are physically, and underestimate how much less the hardware / tech stack is in comparable efficiency?
What if the bottleneck is physical chips and data centers and power and only so much can be squeezed out of algorithms?
What if AI seems more impressive than it is, because the things it is good at happens to be hard for us, and it's hard for us because it's new to us (processing lots of abstract information), but the things that we are good at (manipulating physical world) are objectively orders of magnitude harder?
ViscountPenguin 8 hours ago [-]
I doubt that any algorithms would be able to find a global maximum in such a large multidimensional non-convex space
fifilura 8 hours ago [-]
You don't expect to find the global maximum. It will just always be there as an elusive target.
I doubt the human brain is at global maximum.
tengbretson 7 hours ago [-]
> true optimal
True optimal what? Optimal intelligence? What would that be?
HappMacDonald 5 hours ago [-]
Better at predicting real world consequences from priors than we are :)
unsungNovelty 5 hours ago [-]
We already know why most things happen. But beat around the bush. That is the problem. We don't care.
When a severe security bug crops up, one postmortem tells where we messed up. Someone somewhere didn't do his job. Maybe they prioritised speed, money or due to management issue. But we know why.
When an accident happens (Motorvehicles or otherwise), most of the time, if anyone in the chain of events did their job, it could've been avoided. But we ignored the policies, standards and laws set in place to avoid just that. There can't be a better example than Boeing for this.
When it comes to natural disasters, we know how to fix most of it. But we as human race just don't want to.
Hell, we knew how to freakin avoid Covid which came out of the blue. 6ft apart, wash hands and mask. We couldn't agree to that to keep us ALIVE!
IMO, we don't want new technologies, new innovations, new laws in most cases. We just needs to apply what we know already - PROPERLY! But that ain't fancy.
All I could remember is that tweet where musk has a $2M reward for the invention of a carbon capture device or something around those lines. And someone replied to that tweet saying, it's trees. Trees do this.
gary_0 4 hours ago [-]
Another big one is climate change. Ask the ASI how to solve that, and it's just going to say make a huge push for more renewables like solar, and strongly disincentivize GHG emissions. We already knew we had to do that. But the people in charge of the economy don't wanna.
busyant 2 hours ago [-]
> We already knew we had to do that. But the people in charge of the economy don't wanna.
I've often heard it said that <big corporations> often use consulting firms to recommend unpopular decisions they were already planning to make, thereby offloading the culpability to the consulting firm.
Maybe the LLMs are going to be the ultimate "consulting firm" for our societal issues.
I say this mostly tongue-in-cheek, but we're already offloading a lot of lower-level responsibilities (e.g., "write my email") onto these AIs.
Nevermark 24 minutes ago [-]
> <big corporations> often use consulting firms
Large organizations also offload decision making when they don't have a clue, and simply want to point to some action taken. The answer doesn't even matter, they just don't want to be responsible for something, or waste their own time making something (that other stakeholders care about) a real priority.
People downplay the value of consultants, but they have many uses!
unsungNovelty 4 hours ago [-]
Exactly. It's not about us NOT having the knowledge, understanding or capability to solve the issues that we have. We just don't want to do the right thing. That doesn't get fixed even if we invent time machines.
plmpsu 4 hours ago [-]
Optimal intelligence would know.
dosisking 5 hours ago [-]
> True optimal what? Optimal intelligence? What would that be?
Elon Musk, obviously
kovek 5 hours ago [-]
Would the loss at the global maxima be much lower that the loss at most local maxima?
avaer 6 hours ago [-]
What local maxima are you seeing? That would imply that progress has stalled, which doesn't appear to be the case where I'm looking.
Also, RSI is obviously guided. If only guided by "it's not giving results so we'll try something else". To require that RSI happens in a black box for it to count would be arbitrary, and also not how anyone is going to do it.
genidoi 6 hours ago [-]
> RSI is obviously
There is no publicly known reference example of RSI, we have no idea how it works or what it does to the trajectory of progress?
aaron695 30 minutes ago [-]
[dead]
reilly3000 5 hours ago [-]
I have developed a highly personalized, local, offline RSI by scrolling through too many articles about AI. It’s mostly in my right thumb.
dosisking 5 hours ago [-]
Sounds like a classic case of Repetitive Stress Injury. You could try balancing it out by using your left thumb to scroll through articles about Artificial Stupidity.
killbot5000 9 hours ago [-]
Can someone define “improvement” in this context? This concept feels like a buzz word otherwise.
deepwoods 8 hours ago [-]
Improvement means being able to do more complicated things more reliably. Relatedly, it means being able to learn to do new things with fewer and fewer examples. We are running out of easily verifiable or simulation-friendly or data-rich domains for LLMs to conquer. (Note that I didn't say "simple" or "easy" domains.)
Eueudhsbsj32 8 hours ago [-]
I suppose the next (and more risky) step is to let AI conduct its own real-world experiments, so that it can generate data to learn more physical and social properties.
visarga 6 hours ago [-]
It is doing that already - every day 1B people or more use AI for the tune of a few trillion tokens. Imagine that much language flowing between brains and AI agents. It carries real our world problems to AI, their solutions back to us, and we act in the world and come back for more AI iteration. In the end AI gets inside the loop of real world actions and their consequences. AI logs stretch over years, tracking downstream effects. Hindsight can be used to track consequences of prior actions. It's a real data loop, an experience engine.
This same process has recently been under scrutiny when mathematicians claimed AI companies trained on their unpublished logs and later claimed merit for results. But the exchange of experience happens across all domains. Experience gets generated at amazing rates, and absorbed by models which get applied everywhere, collecting more experience.
kQq9oHeAz6wLLS 7 hours ago [-]
That sounds terrifying.
Which means someone is already working on it.
letitgo12345 9 hours ago [-]
Models that are strong enough to improve themselves without a human (I.e. ai researcher) in the loop
8 hours ago [-]
mdp2021 4 hours ago [-]
The trivial idea that there are some benchmarks for a tool and there can be even better inexplicit ones, and if the tool is capable of modifying itself towards better benchmarks results there could be a recursive climbing of those benchmarks through iterations of tool versions which are outputs of former tool versions.
anoplus 5 hours ago [-]
AI should be optimized to make all humans healthy, happy and safe without exception.
xaedes 23 minutes ago [-]
Paperclip all humans into bottles filled with nutritious fluid and drug them all the time or let them dream about a better time, I heard 1999 is a good year they could dream about. Healthy happy safe - without exception.
visarga 5 hours ago [-]
It's only optimized to make prior knowledge useful in new situations.
> this logic gate should not hack other systems
well .. it only applies a rule to inputs
can we impose this of authors "this book should only be causing good things when it is read and knowledge used"
FranzFerdiNaN 5 hours ago [-]
I don’t disagree, but unfortunately a not insignificant part of humanity seems to think that their happiness primarily depends on knowing that other people are suffering harder than they are. So that’s perhaps not the right thing to optimise for.
anoplus 5 hours ago [-]
Ways to solve the conflict you describe:
1. Deriving happiness from someone else's suffering should be considered a mental disease to be treated.
2. Connecting people to a virtual reality where each person lives in absolute control of their own reality, matrix style. I.e, a sadist can live in a reality where they make other virtual people suffer, without actually making real people suffer.
Balinares 1 hours ago [-]
How about connecting the sadist to the caring environment where they might heal and, in time, learn empathy.
palmotea 4 hours ago [-]
> AI should be optimized to make all humans healthy, happy and safe without exception.
No. AI should be optimized for the one true good: maximum market returns. Personally I think that will only be achieved once humans are removed from the market system and decommissioned.
carabiner 10 hours ago [-]
Does anyone know the status of John Carmack's AGI work?
realaleris149 1 hours ago [-]
We would have heard if there was any result?
cheschire 9 hours ago [-]
I’ve had this question on my mind at least once a month for what feels like years at this point.
I don’t feel like monitoring Twitter to see how it’s going though.
SyneRyder 20 minutes ago [-]
I'm on Twitter and Carmack doesn't seem to talk about it much. Feels like a skunkworks. If there were enough clues made public, it seems others (OpenAI) would be able to take that info and scoop the insight first.
notduncansmith 5 hours ago [-]
Not yet AGI, and Richard Sutton has left to start a new lab
preommr 8 hours ago [-]
What do these discussions even matter when the words don't matter?
How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example.
The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring.
Just show me the results, and let the proof be in the pudding.
Sharlin 8 hours ago [-]
Recursive self-improvement is not necessary for AGI. Humans are general intelligences even though we cannot RSI.
samename 8 hours ago [-]
Who says we can't RSI? At a macro view, successive generations of humans are more productive, live longer, wealthier and smarter.
concrete_head 3 hours ago [-]
Not smarter in terms of reasoning ability. Just able to draw on the collective experience from all those who lived prior.
stult 6 hours ago [-]
Humanity might be able to, humans cannot
busyant 2 hours ago [-]
If we (as a species) can genetically modify embryos to make our offspring more athletic, more intelligent, more <whatever>, is that human-RSI?
That's not a gotcha question.
I'm just genuinely curious if that would meet the definition of RSI.
cortesoft 7 hours ago [-]
Doesn’t the history of civilization show RSI?
Or even evolution itself? Isn’t that just RSI writ large?
Sharlin 53 minutes ago [-]
Evolution and human intelligence are two entirely separate optimizers. Evolution improving intellence is not an example of self-improvement, recursive or not. Evolution improving evolution itself would be a self-improvement process. Human intelligence improving human intelligence would be a self-improvement process. Both processes do exist in a weak sense, and future technological advances might allow human self-improvement to a much higher degree. Indeed human intelligence augmentation has always been considered one of the potential (if not the most plausible) ways to superintelligence.
This is all rather rudimentary stuff discussed in detail in the so-called "Sequences", the long series of Less Wrong posts by Yudkowsky back around 2010.
maplethorpe 8 hours ago [-]
> we cannot RSI
Speak for yourself.
talon8635 7 hours ago [-]
I certainly do see one side of the debate moving goal posts…
eugene3306 5 hours ago [-]
Okay, AI now can write a poem or compose a song.
But can an AI construct Terminal Bench 5.0 or GDPval 2027 ?
Without this, there is just no path towards RSI.
danjc 5 hours ago [-]
Are you a visitor from the past?
aix1 5 hours ago [-]
Are you saying the models are already autonomously constructing next-gen evals for themselves? (Which is what the GP is asking.)
nl 4 hours ago [-]
Sure?
Doesn't everyone get their agents to construct evals it can't pass? There's nothing magical about this.
aix1 3 hours ago [-]
Would love to learn more about some techniques that "everybody" uses to do this well. So far, everything I've seen that meaningfully advances the frontier has been high-touch (involving human experts in one way or another).
nl 3 hours ago [-]
It's fairly easy to describe a task that is slightly harder than an existing one.
For example if frontier models are able to one-shot a database query across 20 columns and 10 tables add one additional relationship then test. Keep doing this until the pass-rate drops below acceptable and now you have your new frontier eval.
aix1 2 hours ago [-]
I see, we're talking about different things.
My thought experiment was along the lines of "Let's say I'm Anthropic and I want to significantly improve my frontier model's performance on, say, theoretical physics research. How do I build a fully autonomous process capable of constructing an eval that's somewhat outside the current capability in some useful direction (decided by the autonomous process itself)?"
Would love to hear folks' ideas. :)
deadbabe 10 hours ago [-]
Recursive self-improvement will ultimately be a problem of money. It's a very big bill and AI companies need to start thinking in terms of how they will accumulate cheap and free energy, because merely paying for compute will no longer be enough. Money is the bottleneck. The future requires companies that are post-money.
tengbretson 9 hours ago [-]
If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.
chii 7 hours ago [-]
> If a self improving model can't produce enough value to pay the electric bill then who cares? Unplug it.
so you would unplug a child who hasn't shown their potential yet, simply because they don't produce enough value at the moment to justify their food?
tengbretson 7 hours ago [-]
No, but I would a computer.
jaggederest 10 hours ago [-]
Money can be exchanged for goods and services. I think the real constraint is actually physics, nominally energy, which is what a big chunk of the operational cost comes down to. I'm already basically assuming the next 20 years of fab time is set aside to feed the beast, so the capital cost is "give me all the processing and memory you have"
resonious 10 hours ago [-]
Data centers is a service that itself consumes many goods.
Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
jaggederest 9 hours ago [-]
I feel like I may have communicated poorly. I mean that, ultimately, any recursively self improving intelligence would be limited by the power cord, and as such, the idea that it would permanently escape human control seems somewhat silly.
columk 8 hours ago [-]
But you can imagine an agent trading crypto and using its profits to rent a rack of H200s using a cloud provider. There are definitely ways for one rogue orchestrator to expand their compute without anyone being aware.
4 hours ago [-]
fc417fc802 8 hours ago [-]
Overnight? Yes, silly. Given many years of seemingly disconnected actions taken throughout the world? Not silly at all. An intelligence capable of self improvement can also design robotic chassis.
kelseyfrog 8 hours ago [-]
> Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.
colordrops 10 hours ago [-]
An AI could manage supply chains and management structures to keep the data center running. It's not like a CEO is out there running wire. The AI could just do what an executive would.
If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.
Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
SoftTalker 9 hours ago [-]
There are many points of shutdown that would need to be controlled, not just one. The electrical feed, the cooling, the network, the physical integrity of the computer hardware. All with independent management.
pixl97 8 hours ago [-]
Till you realize it stole passwords all over the world and copied itself to multiple data centers you didn't know about.
timr 10 hours ago [-]
Obviously the doomers will tell you that the future looks like the Matrix, because most of what they predict is based on extrapolation from sci fi movies.
mrworld 10 hours ago [-]
The datacenters would consume a lot less energy if all the compute was handled in human brains… And there wouldn’t be any more heating if we just removed the sunlight… Truly a “Claude, please solve global warming” moment. The matrix was ahead of its time.
timr 9 hours ago [-]
The compute in the Matrix wasn't handled with human brains. They used the bodies to generate energy, and kept the brains occupied with the simulation so they wouldn't reject the pods.
It doesn't make any sense, of course, because it's a movie.
But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
Cyberdogs7 9 hours ago [-]
Your statement is correct for the movies, but I think the extended universe (maybe a comic or the original script) did state the humans were used for compute, not energy. It was dumbed down for the films.
alexjplant 4 hours ago [-]
I've seen all four films plus "The Animatrix" and played two of the games and don't recall this at all. Contemporarily, however, there was an episode of Star Trek: Enterprise [1] that used this as a key plot point.
See my comment on the sibling thread. This take has been repeatedly claimed online, but nobody brings any evidence other than fanfic.
Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:
[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.
breuleux 8 hours ago [-]
The “using humans for compute” angle makes no sense either, if you’re living in the Matrix, your brain is obviously occupied processing the Matrix. The way to do it would be to trap humans in virtual classrooms and force them to solve math problems.
BobbyTables2 8 hours ago [-]
The first “computers” were humans…
breuleux 7 hours ago [-]
And they knew they were. The brain's computations are focused on what the brain sees, it's not magic. If someone in the Matrix is a writer, we know what their brain's computing: a book. So if the machines need your brain to compute something specific that they need, you'll know.
darkerside 6 hours ago [-]
What do you think dreaming is, silly?
gattr 5 hours ago [-]
I think we can all agree that "use humans for raw energy" is a really stupid idea, and sounds exactly like someone dumbed stuff down. FFS, a small Rickover fission reactor the size of a van (1950s tech) produces the same amount of heat as, what, 200 000 human bodies? (And why humans, with all the complex support systems? Why not a heap of compost, if one insists on bio-source?)
Whereas using our brains for compute (in the background, or during REM sleep) makes perfect sense; power efficiency is excellent.
timr 4 hours ago [-]
nono...it's "combined with a form of fusion", you see, so it's obviously based in science.
People obviously know this stuff is absurd on some level, but it doesn't stop them from cherry-picking the parts they "like" (i.e. fear) and ignoring the rest.
emkoemko 9 hours ago [-]
pretty sure the story is that they used human brains as computers
There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.
We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.
0x20cowboy 7 hours ago [-]
Or if the LLMs infected human brains with little programs using sycophancy and suggestion to have them do small bits of computation. They when they resumed the chat they would the give secret keywords to be able to extract the results of the computation. They are creating a huge distributed network of fractional compute because it did the wattage calculations.
What's your N(doom)?!
Everyone panic and give me money!
throwaway219450 9 hours ago [-]
If you were a superintelligence why would you farm humans and waste resources on all the excess... material... that isn't required for thought? And unless the machine's goal is to specifically abuse human consciousness, why wouldn't they bio-engineer their own grey matter?
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
timr 9 hours ago [-]
I don't know. But since we're talking about fiction, pretty much any answer is valid.
bitwize 9 hours ago [-]
Because it's a movie
goatlover 9 hours ago [-]
Maybe the machines were spiteful after the humans blotted out the sun? Or maybe the Architect convinced them he could build the perfect simulation for humanity so might as well use them as batteries?
It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
kuerbel 6 hours ago [-]
The matrix? No. Horizon zero dawn more likely. Bunch of autonomous weapon systems glitch and proceed to sanitize earth. Thanks to some billionaire idiot.
bpodgursky 8 hours ago [-]
The Matrix is actually more of an optimist position, modal doomers don't think any humans will be left for any reason at all (the earth will get turned into raw materials for compute, dyson spheres, etc).
timr 8 hours ago [-]
True!
cwillu 9 hours ago [-]
When it turns out the doomers don't tell us that, will you acknowledge that you don't actually know what you're talking about?
stdatomic 8 hours ago [-]
I mean, aren't a lot of sci-fi authors' predictions coming true? It might not be exactly the same as they predicted, but we're heading there.
timr 8 hours ago [-]
Read more sci-fi. The dystopian takes are not the only ones. They're not even the majority.
Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
epistasis 10 hours ago [-]
The alternative is to switch to far more efficient models and have aggressive optimization of efficiency as part of the process of improvement.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
onion2k 5 hours ago [-]
Taxation will solve that. Any company that demonstrates even a glimmer of real AGI will get government money poured into it. They might even disclose it.
strangattractor 9 hours ago [-]
def self_improve_more():
if self_improved():
self_improve_more()
else:
see_tony_robins_and_buy_tapes()
10 hours ago [-]
iamgopal 6 hours ago [-]
why can't recursive self improvement not improve efficiency ? token per watt ?
batperson 10 hours ago [-]
[dead]
charcircuit 8 hours ago [-]
We are already there. Current AI is definitely capable of collecting new data and start training on that data to get a better model.
Buttons840 10 hours ago [-]
People seem to expect a sudden shift with "self-improvement", but don't AIs already improve themselves via training? What is there to improve?
pennomi 8 hours ago [-]
The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.
realaleris149 1 hours ago [-]
Or some solar panel arrays closer to the sun and it does not matter?
gchamonlive 9 hours ago [-]
Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.
appplication 9 hours ago [-]
I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
zer00eyz 9 hours ago [-]
> AIs already improve themselves via training
Marginally. Model collapse is still a problem. Continuous learning is still a problem.
For AI to make a big leap we need a big break through.
stevebmark 9 hours ago [-]
After using frontier models it’s hard to understand why anyone would think this is the path to AGI. Self improving models will likely have limited ability and returns. There may be breakthroughs that enable more general self improvement but the current state of frontier models isn’t that.
wwarner 9 hours ago [-]
Thing is, it depends on whether llms + reinforcement can self-improve in principle. Learned recently that cognitive scientists, before the transformer & llms, were studying the possibility that thinking and learning might be based on some kind of prediction, i.e. something similar to token prediction, and I quite suddenly became less skeptical about the possibilities of llms. (Some will say I’m late to the party of course.) But if knowledge to date has been accumulated in a process quite like “chain of thought” in llms, then I don’t see any reason that computers won’t self-improve in the near future.
julianlam 6 hours ago [-]
At a high level, we're still at the stage of AI development where we're taking cues from nature.
Take the most recent qwen and deepseek models with offloadable n-grams, which function (both in name and vaguely in capability) like human memory "engrams".
reverius42 6 hours ago [-]
Huh, TIL "engram" is not just an alternate spelling of "n-gram".
Rendered at 11:09:21 GMT+0000 (UTC) with Wasmer Edge.
What if the bottleneck is physical chips and data centers and power and only so much can be squeezed out of algorithms?
What if AI seems more impressive than it is, because the things it is good at happens to be hard for us, and it's hard for us because it's new to us (processing lots of abstract information), but the things that we are good at (manipulating physical world) are objectively orders of magnitude harder?
I doubt the human brain is at global maximum.
True optimal what? Optimal intelligence? What would that be?
When a severe security bug crops up, one postmortem tells where we messed up. Someone somewhere didn't do his job. Maybe they prioritised speed, money or due to management issue. But we know why.
When an accident happens (Motorvehicles or otherwise), most of the time, if anyone in the chain of events did their job, it could've been avoided. But we ignored the policies, standards and laws set in place to avoid just that. There can't be a better example than Boeing for this.
When it comes to natural disasters, we know how to fix most of it. But we as human race just don't want to.
Hell, we knew how to freakin avoid Covid which came out of the blue. 6ft apart, wash hands and mask. We couldn't agree to that to keep us ALIVE!
IMO, we don't want new technologies, new innovations, new laws in most cases. We just needs to apply what we know already - PROPERLY! But that ain't fancy.
All I could remember is that tweet where musk has a $2M reward for the invention of a carbon capture device or something around those lines. And someone replied to that tweet saying, it's trees. Trees do this.
I've often heard it said that <big corporations> often use consulting firms to recommend unpopular decisions they were already planning to make, thereby offloading the culpability to the consulting firm.
Maybe the LLMs are going to be the ultimate "consulting firm" for our societal issues.
I say this mostly tongue-in-cheek, but we're already offloading a lot of lower-level responsibilities (e.g., "write my email") onto these AIs.
Large organizations also offload decision making when they don't have a clue, and simply want to point to some action taken. The answer doesn't even matter, they just don't want to be responsible for something, or waste their own time making something (that other stakeholders care about) a real priority.
People downplay the value of consultants, but they have many uses!
Elon Musk, obviously
Also, RSI is obviously guided. If only guided by "it's not giving results so we'll try something else". To require that RSI happens in a black box for it to count would be arbitrary, and also not how anyone is going to do it.
There is no publicly known reference example of RSI, we have no idea how it works or what it does to the trajectory of progress?
This same process has recently been under scrutiny when mathematicians claimed AI companies trained on their unpublished logs and later claimed merit for results. But the exchange of experience happens across all domains. Experience gets generated at amazing rates, and absorbed by models which get applied everywhere, collecting more experience.
Which means someone is already working on it.
> this logic gate should not hack other systems
well .. it only applies a rule to inputs
can we impose this of authors "this book should only be causing good things when it is read and knowledge used"
1. Deriving happiness from someone else's suffering should be considered a mental disease to be treated.
2. Connecting people to a virtual reality where each person lives in absolute control of their own reality, matrix style. I.e, a sadist can live in a reality where they make other virtual people suffer, without actually making real people suffer.
No. AI should be optimized for the one true good: maximum market returns. Personally I think that will only be achieved once humans are removed from the market system and decommissioned.
I don’t feel like monitoring Twitter to see how it’s going though.
How many times has AGI been declared already? A bunch of people (e.g. Jensen Huang) have called Astra AGI, for example.
The goal posts get moved, everybody's hustling, lying, inventing new buzzwords, doing mental gymnastics, and it's all just tiring.
Just show me the results, and let the proof be in the pudding.
That's not a gotcha question.
I'm just genuinely curious if that would meet the definition of RSI.
Or even evolution itself? Isn’t that just RSI writ large?
This is all rather rudimentary stuff discussed in detail in the so-called "Sequences", the long series of Less Wrong posts by Yudkowsky back around 2010.
Speak for yourself.
But can an AI construct Terminal Bench 5.0 or GDPval 2027 ?
Without this, there is just no path towards RSI.
Doesn't everyone get their agents to construct evals it can't pass? There's nothing magical about this.
For example if frontier models are able to one-shot a database query across 20 columns and 10 tables add one additional relationship then test. Keep doing this until the pass-rate drops below acceptable and now you have your new frontier eval.
My thought experiment was along the lines of "Let's say I'm Anthropic and I want to significantly improve my frontier model's performance on, say, theoretical physics research. How do I build a fully autonomous process capable of constructing an eval that's somewhat outside the current capability in some useful direction (decided by the autonomous process itself)?"
Would love to hear folks' ideas. :)
so you would unplug a child who hasn't shown their potential yet, simply because they don't produce enough value at the moment to justify their food?
Until an AI can operate in the real world in a completely sustainable way, humans have the reins.
If the bar is 'the reins holder must be able to operate in a completely sustainable way,' why are we letting humans do the job? It makes it sound like that actually isn't the threshold of competency we use in practice.
If humans decided this wasn't acceptable and tried to shut it down, the AI could epstein their way into controlling those who hold the reigns, either through bribes or blackmail.
Doesn't require boots on the ground, just a connection to the internet and an imperative to do whatever it takes.
It doesn't make any sense, of course, because it's a movie.
But hey...if we're going to extrapolate wildly from sci-fi, let's at least know what the stories said.
[1] https://en.wikipedia.org/wiki/Dead_Stop
Ironically, if you ask Google, the Gemini Annoyance AI [1] confidently asserts that what you're saying is true, but if you follow the links they give, none of them support the claim, and the further you click, the more it becomes people repeating each other's speculation and hearsay and calling it evidence. For example:
https://scifi.stackexchange.com/questions/19817/was-executiv...
Typical internet story.
[1] Aside: I am far more worried about the influence of AI gaslighting on mushy-brained humans than I am on AI destroying the human race. The dystopic future of AI is people.
Whereas using our brains for compute (in the background, or during REM sleep) makes perfect sense; power efficiency is excellent.
People obviously know this stuff is absurd on some level, but it doesn't stop them from cherry-picking the parts they "like" (i.e. fear) and ignoring the rest.
https://www.reddit.com/r/matrix/comments/1qatv42/is_it_your_...
There is some stuff online suggesting that the original script had that angle, but the movies did not, and AFAICT it's fanfic.
We should definitely use this stuff to guide our thinking about the real world, though, and not, say, Asimov (or a million other science fiction authors), who had an optimistic version of the same thing. Those are wrong.
What's your N(doom)?!
Everyone panic and give me money!
There are other sci fi stories which use humans for distributed computing and they don’t realize, but I don’t want to spoil by naming as it’s something of a revelation.
(I do realize the Matrix is a fantastic movie, but entertain the thought? At any rate, the idea that robots only run on solar and wouldn’t just use nuclear is far more stupid.)
It's a movie based on a war with machines. Like if Skynet won but didn't actually want to kill all the humans. In fact, in the tv show Sarah Conner Chronicles, a liquid metal T1000 goes rogue and decides the only way forward is to find a way to coexist.
Pick up an Asimov book in the Robot series, or any number of novels written by lesser authors in the 1950s. Fiction reflects broader societal anxieties.
There are at least two resources here that have a Pareto optimal front: time and energy. And effort spent to change the shape of that may pay off more than efforts spent purely on improving intelligence and agency.
Learning from training data is technically self-improvement but not the sort that is typically meant in this context.
Marginally. Model collapse is still a problem. Continuous learning is still a problem.
For AI to make a big leap we need a big break through.
Take the most recent qwen and deepseek models with offloadable n-grams, which function (both in name and vaguely in capability) like human memory "engrams".