I feel like, this is a huge stumbling block that many people have. They'll give a model their data, and ask it questions. I vastly prefer letting the model understand the schema, and then writing functions or programs to answer those questions. I feel like I get way, way better answers. I can have it write unit tests for those functions. I can fuzz test those functions. I can look for data that doesn't fit the schema. I can process new data way faster (and with fewer tokens). I can repeatably get the same answers from the same inputs. I can check the code into a git repo and track changes to it over time. I can share the code with other people. I can review the code. I can improve the speed of the code and get the same answers. I can review the error accumulation, and improve it. I can decide how to handle anomalies, and encode those answers.

It's really neat to see what a frontier model can do itself. No doubt.

But "play chess by hand" is a frankly awful metric. It's kind of like asking someone to take a cube root of some arbitrary decimal, in their head, with no scratch paper.

> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1

The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

the only thing that few people are willing to admit is that humans are the bottleneck as humans are needed to handhold / verify output - which puts a dent or might I say pause on the excessive valuations of a.i companies as that's against the narrative.

Does the LLM need to learn to play chess if it can build a chess engine to play for it instead?

An LLM is the wrong approach for playing chess.

2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.

3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.

Asking GPT to play chess directly using its reasoning only tests its reasoning ability to model chess state. Which it really is NOT optimized for.

This is also true for humans - people who don't have years of chess training can't really tell which moves are legal given an algebraic notation transcript. These people might have good strategic skills in different areas. Chess is just a very, very specific skill

What kind of intelligence is "playing <____> but we don't tell you the rules" supposed to test?

It is also not clear whether the llm adhering to a grammar is necessary for intelligent agents.

Certainly,a harness can easily correct for it.

It can be very interesting and even entertaining to know where models don't do well. I don't find something like chess to be very instructive about anything though, nobody is going to pay for AI to play chess at any significant scale even if it could do it perfectly.

> The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.

I have found that not to be the case, and I am not an AI power user or cheerleader by any means. There's probably a bunch of even "simplest" tasks where AI doesn't do well and might never. That doesn't take away from the cases where it works well and is a productivity booster. It doesn't even have to be solving millennium prizes or any other breakthrough creativity or reasearch, it's still very useful in places.

The caveat is: It depends on the task.

Are there reams of chess moves that the model can train off of? No.

Are there reams of math papers the model can train off of? Yes.

> "current frontier models need laborious oversight and guardrails on even the simplest tasks"

I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.

What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?

Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?

    Array.from(document.body.querySelectorAll('p,li')).filter(e=>e.innerText).map(e=>e.innerText = e.innerText.split('\. ').map(s=>s[0].toUpperCase() + s.slice(1)).join('. ') )

Author's own style is certainly refreshing and welcome over LLM slop that dominates most HN posts now.

If it had proper caps etc, people here would accuse it of written using LLMs.

You just can't win...

It's definitely... unique?

LC;DR :P

What do you mean? The sum of ALL US salaries is $13.4 Trillion per year. According to google $65T is the sum of ALL salaries Globally (not just knowledge workers). It's not reasonable to assume AI is a drop-in-replacement for any job yet (perhaps bottom tier customer support from oversees?).

> So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that

So you're sort of premising here than more than 16% or 1/6 of all the world's jobs get replaced by AI. Hopefully you can understand that's both not the current AI capability and also would be a terrible (unprecedented?) economic shock.

This assumes you don't change the market, but at the scale of (checks notes...) "all knowledge work", that just doesn't hold.

For example if you put 1bn people out of work, you now need some sort of safety net to bail out much of that workforce, a truly unprecedented change. You also lose tens of trillions of dollars of tax revenue.

One solution might be to recoup that cost and lost tax revenue from businesses by raising corporation tax. If corporation tax went from low tens of percent to high tens of percent, would those businesses be able to afford all that AI? No. Same order of magnitude? I doubt it.

There are many possible futures there, but the simplification made in the parent comment is completely unrealistic. The article is right in calling out the valuations as crazy.

That is the after the fact justification of the AGI dollar auction. Each round is kind of 3x the previous cost and neither can really stop because second place in the dollar auction is so much worse than winning.

The only way to stop the auction is one bidder hits a hard budget constraint, both agree to stop, or an outside party breaks the auction.

IMO this is why they want to slow down or have regulation. I think this is also why we see some claims of already reaching "AGI".

The TAM of global knowledge work is just a narrative tacked on after the fact to justify the AGI dollar auction.

The economic fallacy here with the actual valuation is akin to pricing the electric utilities 120+ years ago as some % of the future cash flow of global food production. Take the TAM of global food production and then work back to what % will the electric utilities capture from the advances in the automation of farming? It is nonsense.

The only narrative that actually justifies the capex spend that I can figure out is a first mover AGI monopoly. Even the oligopoly case is hard to justify the capex spend IMO. There is this enormous mismatch between the AGI monopoly and the actual rolling 12-month window of pricing power.

Even the rolling 12-month window of pricing power is going to saturate well before AGI too so it is hard to see how any of this makes economic sense.

And then there’s the second order effect: if all the knowledge workers get automated, who is going to buy the stuff that’s produced?

In one case (financial services) it's thought that expertise is valuable at V=S^2/b4 where V is value, S is skill and b capacity (the leverage available to the manager/expert. b erodes as it becomes harder to find examples of things that are not done well, so if you manage $1bn you might find lots of miss allocations that you can exploit with just that $1bn really effectively, but if you manage $10bn it's much harder to find good places for the extra $9bn. A low hanging fruit effect.

Anyway, that double hit - raw skill and the amount of times you can supply the skill makes the value of skill (V) convex, and it means that in a perfect market (heh heh heh) someone running $100bn is worth 1000's or maybe 10,000's of an average joe expert.

Now, if AI is trusted to run the top 0.1% of everything and has the skill to do it at human top level expertise, then your calc holds. If it's the case that it isn't then more than half of that value disappears. If it's not even top 1% then chop out another 25%.

That implies that we need a lot of trust and a lot of AI capability before these valuations stack up, and it also implies that all other competitors and incumbants are going away. I do not think that Citidal or Bridgewater are going to let Anthropic or OAI take them without a fight. They might lose - but there is a decent bet that they don't. I don't think that many professions like Lawyers or Doctors are just going to roll over and cede their monopoly rights to OAI or Anthropic either.

I don't think that AI companies can charge the same. The human workforce can charge these costs, because of scarcity. But AI systems won't be scarce, it's just a matter of who can run inference cheapest. Plus you still have the human workforce, which might be forced to offer their time for less money.

You need to think in terms of supply and demand.

The demand is there, but the supply is also going to skyrocket. Free open weights models will contribute to supply too.

There will be a new equilibrium that’s hard to predict.

Future supply and demand will set the price - not what is paid today. If supply by open models is vast and cheap, I can't see that the entire knowledge industry can hold the current size. It'll rather collapse to a fraction of its current value.

LLMs are deterministic. They are chaotic, which people confuse for non-deterministic.

I think the core mistake is this partial-equilibrium reasoning. Take the new technology and then hold everything else fixed.

$40 trillion of knowledge work routed unchanged through a new toll booth. Profit. This has nothing to do with reality.

Nvidia on the other hand does have the CUDA monopoly so their toll booth is printing money but that will get routed around or broken at some point.

if that's true, you are wrong.

if that's false, anthropic is dishonest. why trust a dishonest company to be worth anything?

Any reason why that can't be solved through context management and keep-forward scaffolding?

Typically it means knowledge retrieval from a KB or manipulating a control surface not visible to you.

> It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?

We need to have robotics automation catchup first. The math and coding problems are problems in written-space only: you can set up feedback loops to test what worked and what didn't, then try to resolve the defects, maybe back up and try a different path, etc.

What solved coding and maths problems weren't the damn models; open up a chat interface to a SOTA model and you'll see they are pretty limited in producing a solution without a feedback loop.

Instead, it was the harness around the models: it let them explore a space and use feedback to control and direct that exploration.

Until we can do it in meatspace, it's kinda pointless sinking a ton of money into large problems facing mankind...

Like establishing a colony on mars (so the next rock to hit earth isn't an ELE).

Or moving us to a post-scarcity utopia, ending the concept of money.

Or designing and building better batteries for transport that uses only electricity (so that we stop using fossils as fuel).

Or actually building mass-housing. Or mass-farming. Or both, potentially ending homelessness and starvation.

Those are all worthwhile problems to solve, but where's the point of getting a solution on paper? There's no exploratory mechanism there, even for humans, to come up with a solution.

So, all we are left with then is making knowledge workers obsolete: another ELE, but of a different, self-inflicted kind.

AI agents are good at solving well-specified tasks, not at solving problems. They do well in fields where the cost/effort of specification is already part of the business.

That's the thing, it very much does NOT show us that. What happened was mathematicians at openAI learned of an imminent development on this problem, and the insight that it entailed, then they were able to prompt a system in the correct direction and spend 20 million dollars to write down the final steps.

Which is rather precisely the point that the article is making!

> If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer

As the saying goes, if my grandmother had wheels she would have been a truck.

The lack of temperament is very skewed towards the bulls who have been saying AGI is here, software engineering is solved, mathematics is solved, it’s going to destroy the white collar job market, and it’s going to kill us all for like 5 years now.

It is literally denialist about current capabilities

yes.

These companies however are LOSING money (anthropic tries to make it sound like it's profit by deviating from accepted accounting principles) and subsidizing these models. When accounting for all the engineering salaries, training, GPUs, etc, what's their best-case realistic margin three years out, 10%?

So to we'd need a scenario where companies are spending a collective 300B annually on AI (believable) but ALSO that these companies jack up their margins WITHOUT companies switching to the cheaper open-source models (even when there's a $300B incentive to do so).

Putting that aside, prototype software is recombining existing technologies and concepts in well trodden domains, which is distinct from the genuinely novel scientific work the author was contrasting with. Software prototypes are not in the same league, as much as you may like it to be.

Aren't you just describing waterfall? That's still very prevalent in software engineering, and pretty much any other type of engineering – civil, chemical, building, architecture, drug discovery.

It's typically true that software can fail faster and cheaper, but it's also true that the costs are still vastly higher to fix later in the process.

I believe that's obvious - humans don't think in words. Neither do animals. A machine that only thinks in words is obviously going to be deficient in some things, no matter how proficient it is in everything else.

Please share some of these insane things that you speak of..

Some people will complain about the wrong flavours, or missing flavours, or the price, the long lines or maybe it closes early on fridays.

Summarise means different things to different people.