"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
seviu 1 days ago [-]
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
hoppp 1 days ago [-]
Blockchain dapps were getting hacked long before AI.
Not a signal
RetroTechie 1 days ago [-]
If people's pensions & life savings are flushed out the back door (literally), rest assured attention WILL be paid.
somenameforme 1 days ago [-]
I agree with your premise, but not your conclusion. The problem is not the scope or scale of things, but treating them in an absurdly stupid fashion. Why are power systems internet connected? It certainly enables some greater degree of power and flexibility, but it exposes them to potentially catastrophic outcomes in case of malfeasance. So what happened at some point is somebody convinced somebody else that the security was super duper ultra good, and that's simply not true.
And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.
Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.
bagacrap 1 days ago [-]
> Why are power systems internet connected?
Are they?
MSFT_Edging 1 days ago [-]
Yes.
Consider the old joke, "what do you call the medical student that graduated bottom of his class? 'Doctor'".
Same thing applies to sysadmins and IT. Sure there's plenty of skilled people working in facilities out of bumfuck, but also consider how few people want to work in the low-paid, low-respect IT jobs.
Those roles are often filled by the apathetic, mid-skilled, and close in proximity. They're connecting shit to the internet because their boss wants more data points.
betterunix2 1 days ago [-]
Shockingly...yes, and in some cases that is being done intentionally e.g. solar/battery systems require communication within the grid and the Internet is being used as the network.
What makes this particularly embarrassing is that power companies literally run thousands of miles of cables and could easily build a private network instead of using the Internet. The regulations should be amended to require it.
pixl97 1 days ago [-]
Private networks are not that much of a savior... It is incredibly easy to bridge networks intentionally or unintentionally. Stuxnet is just one example. Even private networks have to be as completely secure as a public one. Worse because it's 'private' people think it needs less internal security because "Why secure it, no one can get to it".
betterunix2 1 days ago [-]
The military's approach to that problem is to use color-coded cables and ports, and we already use things like 802.1x to prevent unauthorized devices from being attached to corporate networks. This is not a novel or unsolvable problem.
pixl97 1 days ago [-]
>prevent unauthorized devices
Again, STUXNET shows that bugs get into authorized devices really easily. One of the biggest issues is you have power guys that don't know dick about computer security using fleets of laptops that will need a huge amount of training to bring them up to the needed high levels of security.
More so there are thousands of these devices out in the field in power company trucks. All it takes is someone picking one of these up and infecting it for the problems to spread.
It is an insanely hard problem to solve. Breaches happen in military networks pretty often. As a measure of secrecy, they never tell you about it.
betterunix2 21 hours ago [-]
I would not call STUXNET proof that this can be done "easily" -- that was joint work by the NSA and Israeli intelligence, who are among the best hackers in the world, and it used 4 separate 0days simultaneously.
While you certainly make a good point about the difficulties that come with securing the laptops and other equipment, the military has the same problem. Most of the users of classified military systems are not themselves experts in computer security. The necessary training is not a graduate level course in security; it is training people to always match cable colors to port colors, to only use laptops provisioned by the organization, etc. The military disables, sometimes by physical means, USB ports on laptops that are used on classified networks to close one of the key security holes Stuxnet took advantage of.
Sure, it is challenging, but this is not impossible to solve. Utilities would have to be forced to do it and they would need time to overhaul their operations, but they certainly can do it, and there is an argument that the NSA's IA division should help (as the electrical grid is a critical for national security). It can be done, we just have to be willing to do it (and pay the cost of doing it).
Oras 1 days ago [-]
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
bemtevi 1 days ago [-]
Would it? I don't think so. Not if the weapon is an indiscriminate one and makes everyone worse off. Yes, some people (maybe military generals) might support it. But there would be near universal hatred for it, if people wake up one day and find their bank accounts not working or their eletricity not working.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
zombot 1 days ago [-]
> Not if the weapon is an indiscriminate one and makes everyone worse off.
Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.
bemtevi 1 days ago [-]
> Nuclear weapons are indiscriminate and make everyone worse off if used.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
zombot 1 days ago [-]
"most people" are in the set of things the trillionaires don't give a shit about. Their market distortion has already made PCs unaffordable and electricity prices rise, among other adverse effects. What else has to happen so you take notice?
Oras 1 days ago [-]
To add to your point, many people’s jobs had been affected by AI, job market data don’t lie. That didn’t stop the progress, it became exponentially faster.
pigpop 1 days ago [-]
Right, just like the war to end all wars totally prevented the nations involved from going to war with each other... for a grand total of 20 years before they decided to do it again. There are other more powerful and influential factors that decide the fate of nations. Even if you get your wish and everyone suffers and everyone hates AI then it will simply disappear into classified research projects and government run data centres. The NSA could easily absorb it but I'm guessing government contractors would also want a piece of the pie for weapons research.
If we're at the point where all that matters is scale and reinforcement training then the public facing aspects could entirely disappear and progress would still continue especially when given an infinite budget of taxpayer dollars.
andrewflnr 1 days ago [-]
It would at least increase the caution and regulation around AI, which would be beneficial.
goda90 1 days ago [-]
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
_heimdall 1 days ago [-]
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
graemep 1 days ago [-]
It would be a lesser evil rather than good.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
_heimdall 1 days ago [-]
One can justify doing terrible things when the ends justify the means, in this case when learning a lesson sooner than later justifies all the side effects of a long power outage or similar.
The OP didn't say we should make it happen so I'm not intending to put those words in their mouth. Its a small leap from "it might be a good thing if the power is out for days" to someone hearing or thinking that themselves and acting on it.
If it turns out we have a power outage, people die, and we learn a lesson and adjust course then so be it. That would be what happens in that future but it isn't necessary and it isn't a good thing even if today we think it leads to the best outcome - we can't know that without knowing all the outcomes and would be generalizing into only the ones we can imagine.
graemep 23 hours ago [-]
I agree about the invalidity of the end justifies the means arguments. I hope no one acts on it, but if (probably when ) it does happen and is only something like power out for days in a limited area it will be true that it could have been (a lot) worse. How much worse it could be is terrifying.
bemtevi 1 days ago [-]
> people will die.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
_heimdall 1 days ago [-]
I'm never a fan of an "ends justify the means" argument though.
We both don't know all the possible futures and we don't know if a power outage or similar would kill 1,000 or 1,000,000. We also don't know if that loss would lead to the future we want.
djdieie38485dj 1 days ago [-]
I’m old enough, and he definitely is, to remember when having something connected to a network was a considerable security consideration
Now supermarket fridges are openly accessible over the Internet
I’m not sure it’s AI that’s the issue, it’s just a scapegoat for decades of bad security choices that nation state bad actors could have always exploited
Now that’s a perceived threat, ‘AI will do bad things’ is a great excuse for awful national infrastructure and security management
tpoacher 1 days ago [-]
To me this sounds a bit like "a zombie apocalypse wouldn't be such a bad thing, because natural evolution would lead to humans becoming faster runners" line of reasoning.
sailingparrot 1 days ago [-]
You are missing OP’s point.
A temporary blackout is specifically not an apocalypse, it’s temporary. But it might be painful enough to jolt us into action. As opposed to have the first major AI catastrophe be an apocalypse right away with no chance for human concensus to form beforehand.
hardbass 1 days ago [-]
The one thing worse than security by obscurity is probably security by fiat.
dust42 1 days ago [-]
So Hinton doesn't trust the tech leaders but then the consequence of having governments censor LLMs is that only tech leaders and governments have unlimited access to AI.
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
lp92 4 hours ago [-]
> If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand
Exactly. This is the same type of sentiment as security through obscurity that Hinton is advocating for.
api 1 days ago [-]
I would posit that systems being hacked by AI were already vulnerable to the huge caches of secret zero days held by intelligence agencies and organized crime around the world. Which we know exist.
Insecure is insecure. These systems never were secure.
hardbass 1 days ago [-]
I am not sure what people are trying to get at by claiming "banning" AI is a solution to security bugs that always existed but are just now being uncovered. Security by obscurity at least has some logic, security by fiat is absurd and meaningless.
vonneumannstan 1 days ago [-]
If you are stuck on censorship you are just way outside the window of what is actually of concern here...
bagacrap 1 days ago [-]
I think the point is that we can be oppressed by governments or corporations using this super human tool just as well as we can be oppressed by skynet. Presumably the only way we could protect ourselves is with our own un-neutered AI. Or something.
bvan 1 days ago [-]
Of course, we are.
Back in the late 80’s early 90’s, no one working on ML was aspiring to develop systems which would be used to generate the kind of AI crap which plagues us today.The good is being smothered by the absolutely soul-crushing pointless.
lagrange77 1 days ago [-]
I hear you and i find it devastating, too. On top of that, the negative effects on the planet for having this technology vomit this soul-crushingly pointless crap is just depressing.
api 1 days ago [-]
Before AI I was already being smothered by soul crushing pointless junk made by humans.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
hollowturtle 1 days ago [-]
Like I said on a comment above I think you're wrong because:
> Human slop is slow and easy to contain and build containments around it. Generative AI is so fast that it breaks every possible containment strategy
nater5000 1 days ago [-]
We have been living in a "late-stage capitalism nightmare" for longer than generative AI has been around. We've been on a path towards destroying the world via climate change explicitly because of "human slop" that we could not contain. People's brains have been rotting because we're a few generations deep into mass media producing endless human slop that people are addicted to consuming, and there's been effectively no containment around it.
If you think society prior to ChatGPT was any less off-the-rails than it is now, then you're suffering from recency bias. I'm willing to give you that AI certainly accelerates the slop, but it doesn't fundamentally change anything about humans being unable to regulate their consumption of slop.
watwut 1 days ago [-]
"algorithms and social media incentive systems" are run by the same group of people and are crap for exact same reasons.
But also no, the crap AI is causing goes waay beyond algorithms and social media. It is ok to blame it.
kajumix 1 days ago [-]
AI exposes the Sturgeon's law (90% of everything is crap), and amplifies it much more, and anything good is drowned out. Most of human-generated popular music is slop. Most of human generated code is buggy, unmaintainable and full of debt. Social media and youtube have always been dominated by click-bait and mediocre time-wasting crap.
hollowturtle 1 days ago [-]
> Most of human-generated popular music is slop
I think all these laws/theories etc needs an adjustment to today's reality. Human slop is slow and easy to contain and build containments around it. Generative AI is so fast that it breaks every possible containment strategy
pixl97 1 days ago [-]
It seems like a lot of people forget that quantity and speed are qualities.
A snowflake is not a problem.
A lot of snowflakes can be an issue as the pile up.
A lot of snowflakes rushing down a mountain at 100 miles per hour is a big damned problem.
rickydroll 1 days ago [-]
My life was not better without AI. It's my accessibility tool. Speech recognition is far better than anything Dragon produced (I use Aqua). And I can now write code again. It's been 30 years since I've been able to do anything significant without pain. It is incredibly liberating to be able to finally write what's in my head and make the code function.
Having AI in my life is a hill I will die on. I will not want to go back to being dependent on having other people do for me what AI lets me do for myself.
reasonableklout 1 days ago [-]
Ok, but Hinton is worried about what happens if the tech keeps getting better at the current pace.
Let's flip your last point: if the tech gets good enough to replace people at any task (including physical ones), do you think others will need you for much?
And then there's the systems that the corporations are making more powerful as fast as they can. Yes, they promise to do things like cure cancer, but they're also capable of scarier things like hacking banks. And they're progressing faster than we can understand them. Would it be so bad to delay the cancer cures by a couple years to ensure we still understand what the models are doing?
rickydroll 1 days ago [-]
I don't care if others need me, and here's why.
With AI-driven speech recognition, I have far more agency than I did without, period. I can write, I can code, I can communicate with various entities, people, government organizations, corporations, chatbots. I can fill in online forms instead of leaving a bunch of spastic scrawls that generate more questions. I can use it to drive OpenSCAD and 3d print something.
And here's why I start out by saying I don't care if others need me.
I think I'm congenitally incapable of not building/designing/exploring things. If we had an AI-dominated future and we were all on UBI, I'd still find myself saying “hmmm, I wonder” and going down a rat hole to see what I can find. I'm doing it for myself. My curiosity and satisfaction in bringing something to life, even if it's only a variation on what somebody else did.
Rephrasing something I said in a comment somewhere that I would spend time gardening, bird watching, and building a closer relationship with my partner and my pets. I would have no want of things to do or explore.
To your last point, we don't need to understand how something works for it to be useful. Doctors don't understand why treatments work and why they fail. If doctors waited, if treatments waited until they understood how and why it worked, I'd probably be dead. But because they don't, I now have a VP shunt in my head (aka, medical-grade juice-box straw), and my life has drastically improved. Toddlers are something parents don't understand—the how or why they do what they do—but they keep having them anyway.
WTF moments are an emergent property of toddlers, goats, border collies, working dogs, hunting dogs, and AI agents. And like with a toddler, we need to deal with it like we're the adults in the room.
Too many people in my life have died of cancer, and the agony and pain as their body hollows out to a waxen shell is something I'll probably never forget. If I had a choice between understanding how AI works and a cancer cure, I would choose the cancer cure, hands down.
One thing we do need to do is slow down the development or the build-out of AI data centers and put a resource constraint on the model companies, forcing them to get better with the same resources. Sort of an economic “you can't have dessert until you've eaten your vegetables” moment.
I really got to sit down and write that blog post on why agents are like toddlers and that toddlers should have figurative machine guns.
abandonliberty 1 days ago [-]
I think you're missing the point here.
If one doesn't provide incremental value, how do they get food, shelter, and compute?
Perhaps we'll be lucky like horses, and be lucky enough to become pets once our population collapses.
rickydroll 21 hours ago [-]
Maybe not by as much as you think.
Without accessibility tools, there's not much incremental value I can provide. When the RSI showed up, I was advised by many people to spend the rest of my life on SSDI. I refuse to do that, but it was difficult to show I could provide incremental value. Yes, my mind still worked; the designer, the architect were still there. I had speech recognition so I could write, but a common reaction was, “How can you do anything technical? Your hands don't work.”
Now to the general case that you seem to be worried about: that AI will remove everybody's ability to provide incremental value. No worries there. The entire system will collapse. Labor has no value. Without labor, capital has no value. And we're all jolly well fucked. Now, Your value is how good you can become at asymmetrical class warfare.
At least that's where the current nihilist fantasies about AI leave you.
The AI nihilist view works when you have the wrong enemy. AI is merely the puppet, the distraction, just like the H-1B visa, offshoring, undocumented workers, etc. All these distractions are so that we won't notice the billionaires are enshittifying society and won't seek to dethrone them.
IMO, I argue the fix involves returning the tools of production to the hands of labor, and taking power away from capital. But as with many problems in life, the solution is left as an exercise for the reader.
reasonableklout 19 hours ago [-]
I'm confused, why does capital have no value without labor? I thought the whole point of AI is to enable capital to fully replace labor.
I sympathize with "dethroning the billionaires" but to me it looks a lot like powerful, general AI systems are the most capital enhancing technology we've ever seen. Wouldn't the class struggle benefit from slowing down frontier AI development while diffusing local models to ordinary people? I feel both can happen at the same time.
rickydroll 9 hours ago [-]
Those are good questions. IMO, capital only has value when it can be exchanged for an asset. Assets are the direct result of labor or hallucinations, e.g., a brick or a share of stock. If there's no labor, there are no more assets produced. If there are no assets, the value of capital goes down. Occasionally I think the endgame is reducing everyone to the oldest and second oldest professions.
As to the point of AI, it's fuzzy. We have a tool we don't understand how to use; it's incredibly valuable in some spaces, destructive in others, but we really don't know how or where to use it. We are trying to fit it onto the old model of work and labor value creation but don't yet understand what the new model should be. It's analogous to generals fighting the last war.
General AI systems: capital enhancing? I can understand how you and many others see that, but right now it is capital-consuming. Capitol is desperately looking for that model they can wield to get more capital out of the system than they put in. With the opinion reports describing frontier AI companies' bond offerings as junk, it looks like asset gamblers with capital are trying to unwind their exposure because they don't see how they're going to get returns.
I suspect the meme of using AI to reduce or eliminate labor is a continuation of capital's hostility to labor having any power at all. It has traction because capital sees reducing labor costs as a way of extracting value from their investments, and, as I said, they can't see any other way to do it.
Slowing down development won't make a difference to the class struggle. There are many parties in this race, and handicapping one does not slow down the others. However, slowing down deployment, taking time to figure out the right way to use it would greatly benefit the class struggle. Denying AI companies' data centers is probably the most effective way to slow them down. If they don't have the resources to spend tokens on answering your email, they will find more valuable ways to spend tokens that should benefit all of us.
Denying them data centers will free up resources, which will enable us to create distributed AI use. Altman's view that all the RAM belongs to him because it's more efficient is a valid model if you are trying to fit AI into the way we used to do things. Progress and new economic development seem to emerge from the edges and not the central power structures of an economy. So if we want to find that new model of how to work with AI, it is imperative that we can build out local models in the hands of many people.
insane_dreamer 18 hours ago [-]
That's good. And I'm glad for you. But honestly, you're an outlier, or at the very least on the long tail of the distribution.
rickydroll 9 hours ago [-]
Yeah, that's the excuse TABS has used for ages to deny disabled people agency and that they have any value. You might want to keep in mind that as you pass through the stages of life, age and accident will remove ability, and you will need these tools to keep doing what you love and are good at.
andai 1 days ago [-]
> temporary blackout of financial or power systems
I'm reading this in the context of the article currently at the top:
AI is forcing us to pay off 40 years of technical debt
Is it bad we're finding all the bugs now? Or is it good that we're finally being forced to make our critical systems actually robust?
keeda 21 hours ago [-]
I would say: Like all things AI, very bad in the short term, but very good in the long term. Unfortunately, it's us who have to live through the short term.
I guess we could look at it as going through a world-wide Code Red...
wewewedxfgdf 1 days ago [-]
Speak for yourself - I'm in AI heaven.
Programming has never been this awesome.
Janicc 1 days ago [-]
Yeah, seriously. AI has been the single biggest technological improvement to my life in the last 20 years. If I could choose between smartphones and AI, I'll take AI any time of the day. Smartphones (in combination with social media) had a worse impact on society than AI has had and are not worth their advantages to me.
m11a 1 days ago [-]
Yeah, it's single-handedly one of the best inventions this field has ever had.
1 days ago [-]
geraneum 1 days ago [-]
The people in wall-e were in AI heaven.
tomjuggler 1 days ago [-]
Can't agree more - some of us are just happily achieving our life goals in record time thanks to AI!
guitarlimeo 1 days ago [-]
There's just something off about this mindset - why is achieving your life goals faster better? I think you haven't realized that life isn't a bunch of checklists you need to complete but rather about the journey, and when you do get that, you don't need to go fast.
Also the future where there is no sacrifice to achieve your lifegoals seems just unnatural to me.
CrimsonRain 1 days ago [-]
I can type two prompts to solve a task while I spend time with my little boy. Get notified in phone, reply using voice and off AI goes to fix remaining issues.
I am building personal software for family (home automation where it is not possible, time capsule (future letters for kids), study materials for kids and much more.
Alternative is, it would take me hours of manual work and I'm stuck at computer.
Yes, I'm in AI heaven right now that allows me to spend way more time with family than I would have been able to and still be productive.
guitarlimeo 15 hours ago [-]
I wouldn't categorize any of those as a life goal but you do you.
Cordiali 23 hours ago [-]
Some things in life need to be treated as tools to accomplish a goal. We don't have the mental capacity to develop an understanding of everything in the modern world, so we specialise instead.
It's no different from someone seeing their car as just a tool to get from A to B. That might be an understandable mindset for people on HN, whereas people at a car meet might find something off about that mindset.
Not to mention, sacrifice can come in many forms. The simplest example is, the time required to learn something or develop a skill. As long as people want to pursue things just out of interest, that's not going away.
(Unless we find a way to download the information directly into our brains, I suppose.)
hollowturtle 1 days ago [-]
Show don't tell
bigstrat2003 1 days ago [-]
Having a machine write code for you isn't programming, any more than hiring devs to write software for your business was programming.
archagon 1 days ago [-]
"NUH UH!"
Thank you, brave HN commenter. You've truly made the site a better place with your astute rebuttal.
gchamonlive 1 days ago [-]
“I think [Mark] Zuckerberg and [Elon] Musk and Sam Altman, they understand the risks, and they don’t give a shit. They’d rather have the money. That’s my belief.”
What's risk for people valued close to or over trillion dollars? Risk is meaningless when corporations get too big to fail.
rglover 1 days ago [-]
> What's risk for people valued close to or over trillion dollars?
Mortality.
gchamonlive 1 days ago [-]
Does it increase the more business risk these folks take?
rglover 1 days ago [-]
I would guess for some, certainly. Acknowledging that life is short and that, best as we know, you get one shot likely animates a lot of the seemingly frenetic behavior and risk taking. They don't want to die before they've 'made their mark' (whatever that may mean to them).
gchamonlive 1 days ago [-]
Statistically I mean
jrootabega 1 days ago [-]
That's weird, they must have cut off the part where he said Sundar.
socketcluster 1 days ago [-]
My main fear is that the integrity of everything will be compromised, but everyone will keep acting as if it's not. I feel like it's already somewhat the case but I'm worried about it getting worse.
My prediction is that some people will be able to use AI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares (or crypto tokens) of the top company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
NoidFonsense 1 days ago [-]
I suppose, but precious metals and cryptocurriencies would still exist, and would probably be more immune to power struggles and hacking than stocks.
Not to say that they're unhackable or unstealable, but merely that they dont have a central weak point that if broken, destroys the whole system
socketcluster 1 days ago [-]
I also think so. The weakness of current popular cryptocurrencies though, is the complexity and size of their codebases. Many above 100k lines of code. That's risky. Though the benefit is that they are somewhat heterogeneous (different nodes running slightly different code/versions) so it may be hard to find a vulnerability which affects more than 50% of nodes; and this is a hard requirement for compromising the integrity of that system.
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So yeah I also think crypto will win in the long run. I would bet on a tokenized stock with AI attached.
I think the top asset will combine the AI which is best at hacking with the blockchain which is hardest to hack.
Ability to corrupt others whilst maintaining perfect integrity yourself.
AI for offence, blockchain for defence.
gchamonlive 1 days ago [-]
As is the case with any technology in the hands of careless people, but unfortunately we can't wait for society to catch up morally to the technical standards, innovation is organic and breakthroughs have no timing.
gkcnlr 1 days ago [-]
> [...] as he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems.
I don't buy this. There isn't some kind of voodoo "rogue AI(s)" out there which could randomly decide to take over their vendor stack or brute force their way coming into possession of the most critical infrastructures on earth. These kind of predictions could only be realized if a tiny minority who owns a huge amount of compute and a particular agenda decides to do this kind of thing, which also will require insane amount of data center capability to cause, especially some kind of financial "collapse".
pixl97 1 days ago [-]
You mean like OpenAI training their next model and not monitoring it?
Or some other government setting an AI to intentionally hack and crash the US?
>possession of the most critical infrastructures on earth.
I think you may underestimate how poorly we protect these things. They have all the same security flaws as our internet connected systems. They aren't written in some super special language, it's all C, C++, JS and other non-memory protected trash.
feverzsj 2 days ago [-]
AI winter has arrived.
yewenjie 1 days ago [-]
Said AI "experts" for the last 10 years.
emil-lp 1 days ago [-]
Nobody has said that. The winter will probably come, but it's not at all clear when. First the bubble must burst.
1 days ago [-]
andsoitis 1 days ago [-]
Nobody said that? The GP literally said it!
slibhb 1 days ago [-]
It might turn out that way. Or it might turn out that we're better off with AI. There are plausible arguments either way, and so the debate comes down to vibes. Essentially it's "I'm an optimist" vs. "I'm a pessimist."
I decided long ago that - when there are plausible arguments either way - it's better to be an optimist. Optimism does not mean ignorance or denial; rather, optimists should acknowledge risks and problems while having confidence in our ability to generate solutions.
One of the biggest challenges facing us is a critical mass of pessimists. Not only regarding AI, but also with respect to the economy, democracy, liberalism, etc. And while these negative views are plausibly correct, they also act as a self-fulfilling prophecy in that they rob people of the agency that is needed to solve problems.
strideashort 1 days ago [-]
We were probably better off before agriculture, but here we are.
93po 1 days ago [-]
i mean personally, i enjoy not having smallpox
1 days ago [-]
amai 10 hours ago [-]
Oppenheimer: it might turn out we were better off without nuclear bombs.
informal007 1 days ago [-]
Calling for Off-gird mesh communication
vnlamp 1 days ago [-]
The same applies to most of humanity’s other inventions, such as transportation, weapon, electricity... Ultimately, these technologies serve as instruments for rivalry, deciding the fate of the strong versus the weak.
1970-01-01 1 days ago [-]
AI would not survive a Carrington event and that's really the only threat to it at this point. Even WWIII would directly take it into the future.
beej71 1 days ago [-]
A glorious abundant future is the harder of the two targets, since it's not what the people building the models are working toward.
> It could help cure diseases, end boring work and give us more free time.
Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.
_heimdall 1 days ago [-]
"Could" does a lot of work in these kinds of claims.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
bemtevi 1 days ago [-]
> We need to build it as fast as possible because it could cure all disease
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
_heimdall 1 days ago [-]
They believe they're building a magic genie. If one holds that view, it seems reasonable to make nearly every trade off to get there as you can do anything you want once you have magic in a bottle.
staminade 1 days ago [-]
How has AI given you less free time?
bemtevi 1 days ago [-]
I have to spend extra time just trying to find AI-free things, AI-free writing, AI-free communities, and people who aren't addicted to AI. And get noticed/get a job by having to push through all this AI static. And having to interact with AI-based customer service agents that take 10x as long to resolve the problem as a human being. And having to wait a bit longer because there are 10x more 'Are You Human' Captchas slowing down website loading. And having to download OS updates that are 2x as large because they contain AI models. And having to double-check everything because there's so much more hallucinated crap out there.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
AI is shit.
cat-snatcher 1 days ago [-]
ai psychosis
bemtevi 1 days ago [-]
I really don't know what you mean because I don't use AI at all...
perarneng 1 days ago [-]
That's the whole point of the site. Did you read it. AI should not cause harm, direct or indirect.
bemtevi 1 days ago [-]
Of course I read it. But my thesis is that one cannot avoid that. Causing harm, whether direct or indirect, is the fundamental nature of AI.
hardbass 1 days ago [-]
Causing harm, whether direct or indirect, is the fundamental nature of humans.
perarneng 1 days ago [-]
It can be avoided, humans control AI still. Humans are the ones causing harm with the help of humans.
andsoitis 1 days ago [-]
> Causing harm, whether direct or indirect, is the fundamental nature of AI.
How so?
233mhz 1 days ago [-]
I mean, it could if "we" "just" got out of globalized financial capitalism, changed everything about our political, economical, financial, social systems, etc.
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
1 days ago [-]
bemtevi 1 days ago [-]
I don't think accelerationists think we can do anything. The thought process of the accelerationist is partly fueled by the 'excitement' of betting and playing musical chairs and hoping to be the first ones out holding a bag of money and partly fueld by the delusion that technological advancement is always good at any level of complexity.
koe123 1 days ago [-]
You’re a cultist
perarneng 1 days ago [-]
How did you come to that conclusion?
jdw64 1 days ago [-]
To be honest, I think it depends on the environment. In the past, if you didn't come out of a good university, your access to knowledge was limited. But these days, in terms of access to knowledge, I think things have clearly improved, whether or not AI is illegally sharing knowledge.
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
rhines 1 days ago [-]
I'm not sure of this. I remember back in the day finding people's blogs, reading old forum threads, participating in said forums - there was a ton of info out there and easily found. Now those kinds of sites are rare, and those that still exist aren't surfaced by search engines like they used to be.
AI has the benefit of having trained on all that material, it's a snapshot in a sense. But as people increasingly stop creating content like that and we turn to AI instead of asking for help on forums we stop growing the knowledgebase that AI draws from.
jdw64 1 days ago [-]
I don't know whether it's nostalgia for the past or whether it really was easier to find things. If you say so, then it could be. It's true that in certain fields it really was easy to find things, and it also differed from country to country. Where I was, there was mainly gatekeeping—a kind of control. In that sense, was the past really better? I don't know. In the past, there were quite a few cases where learning a particular skill required an absurd amount of money (though this may be because I'm from an East Asian country). I think what made people enthusiastic about Udemy was probably that, in my memory, it offered those expensive courses cheaply or gathered the information together.
I don't think forums are bad, but I've questioned whether experiences in such forums were always pleasant. There was a lot of wrong information, and there were also many people who picked fights for no reason simply because they didn't like something. But AI really does just tell you knowledge easily.
Of course, forum activity gave me the opportunity to come across knowledge whose keywords I didn't even know, and that got me studying, but I question whether such forum activity was open to everyone.
I also somewhat agree that the problem of the knowledge base shrinking is real. But is that problem really unsolvable? As far as I know, the results on the degradation problem of synthetic data say that if there is synthetic data, there isn't that much degradation. People, too, tend to idealize the past as they get older. Like treating the inconveniences of that time as nostalgia.
I don't know. Rather, once the current SLOP era passes, certain knowledgeable people may become rare and be treated well then. The future can't be known.
Anyway, this is purely a matter of values
"Some people may regret the destruction of knowledge communities, but I benefited from that destruction, so I can say it's good. People always judge by their own experience, don't they? I respect your past experience. In the cases you went through, I think AI produced more bad effects than good.
I've also seen many things disappear because of AI, but it let me try things I wanted to try, and I think that part is good. So, honestly, I don't know. I have both affection and hatred for AI. But the affection seems a little bigger.
rhines 8 hours ago [-]
Yeah it probably did vary by field and by country. Also when you bring up Udemy I think that might illustrate a difference in perspective that's relevant - I'm thinking of finding very specific info. Like searching for an exact implementation detail of something, giving exact keywords and wanting results that match what I typed. That functionality just doesn't exist anymore, at least not with mainstream search engines. Udemy however is more about teaching a general topic, and I think this is something where AI helps a lot - Udemy is essentially trusting a teacher to summarize a large topic into its key components, and AI can do that quite well for a lot of subjects.
And certainly dealing with people online can result in unpleasant interactions. I found that specialized forums were usually well managed and wouldn't tolerate abusive members, though new members were expected to learn the forum's rules and put some effort into finding existing info before they start asking questions. The downside of AI is that it can only tell you things that are already visible online, and it will take the stance of the majority. On a forum you would get people who can chime in based on their experiences where there's absolutely no prior info online, or people who could contest the majority and explain why.
Synthetic data certainly could solve this. Especially if we get AI controlled robots or AI managed physically accurate simulations of the world, such that AI can actually engage with the world and test new things. It will take a long time before we get to that point where AI is able to reach the level of being able to tell you something like "oh yeah if you take apart that DSLR lens, the screws holding the floating group in place are glued in with loctite. And for good reason, if you don't have an optical bench you're never going to be able to get those elements recalibrated if you adjust them. But it looks like your copy has a shim behind that group - try removing that and see if that lets you focus past infinity." Like AI can understand a lot of that in the abstract, and might be able to guess at some of it, but until it's actually pulled apart this lens and messed around with it it's going to be vulnerable to just making things up or simply not knowing key details. But that's why it does so well with code, since all code is accessible to it.
I do agree that AI is useful though, and I have no qualms about using it to learn things faster, build tools that help me, or do my job. And though AI has contributed to the loss of the internet I value, it's not the only contributor, and likely not even the main contributor. Search engines were already consolidating power into a smaller subset of large commercial sites rather than surfacing forums and blogs, sites naturally went offline over time and archiving efforts just weren't sufficient to make up for this, and users just prefer lower effort forms of engagement like scrolling Reddit with everything in one place and lots of entertainment rather than being a member of multiple more focused communities. So while I may malign AI for the impact it has had, you're probably right that on this topic the net impact is positive.
jdw64 6 hours ago [-]
Thank you for the in-depth comment. We have some points where we disagree, but also points where we're similar, and I think this has been a genuinely enjoyable discussion. I largely agree with your current comment.
I also agree that specialized forums have well-managed aspects. But conversely, getting into a specialized forum usually required a ticket of admission—something like having to attend a certain conference socially.
So the problem is this point. Very high-level forums required an admission ticket, or basically a certain level of qualification, and were hard to get into; you had to get in through an existing member, and if you were physically far from that member, it was hard.
Usually, the forums someone like me could get into were often ones at a somewhat lower level. These days things move to Discord, but it's hard to go into Discord's closed ecosystem and study there.
Actually, as you said, forum activity and human interaction are important. Because in the end, humans have to learn about the world through humans.
The world isn't simply a place to obtain information—things like the patience to endure unpleasant situations and people, the understanding to listen to others' opinions without distorting them, and the courage to admit to others what you don't know are not things that can be cultivated through AI, I think.
The various values that have limits when learned through AI can be learned by mixing with people. In that sense, people have to learn about the world among people.
Because AI basically gives you wrong information to please you and reinforces information only within the scope of what you already know. I think friction with people is necessary to find keywords you genuinely don't know.
My words got sidetracked in the middle. I agree with your comment that AI isn't the only thing that changed the internet this way, and I also agree with your point that forum activity and the like are necessary. It's just that where we put the emphasis and weight is what makes our thinking differ. Thank you for the long comment, and have a good day
TonyStr 1 days ago [-]
Knowledge was already extremely accessible in the age of the Internet, but it required time to read and learn. AI is making that same knowledge applicable without the prerequisite of learning.
Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?
jdw64 1 days ago [-]
I don't think you're wrong, but fundamentally we're in an age of information overload, and this may be the new normal. In other words, the environment where you had to know things deeply and become an expert, like in the past, may be different from an environment where you know many things broadly and combine them.
I basically don't think you're wrong. However, books and reading have their information fixed at the point of publication, whereas depending on where you live or your community, the information you get buried in differs. Because I was in an Asian country, it was actually very hard for me to synchronize my programming skills with the latest technology in the US. The frameworks commonly used in my country are stuck at roughly 15 years ago. The time lag between US technology and my mainstream technology is about 5 years. In my area, it's about 15 years.
Anyway, thinking about your opinion, I think there are both valid points and dark points. I think learning for its own sake is important.
I agree that people should have an attitude of learning for its own sake, but I don't think it's mandatory. I think it should vary depending on the environment. Still... I do think having that attitude is a good thing. I like your idealism, senior.
However, that holds for people who are middle class or above and can survive. Because rich kids can picture the future and are given the right to love learning. But if you're poor, that's very hard. And when you grow up in such a poor family, there's an emphasis on it having to lead to practical skills.
And even if someone loves learning for its own sake, I think quite a lot of people love the honor that comes from learning. But that too can be called loving learning itself. It's purely a matter of individual values.
I don't think you're wrong, senior, but I also wonder whether the old ways are right in today's era of excessive information. It's just that because our lives and the environments surrounding us are different, we came to different conclusions. I upvoted because I agree with your thoughts. But there are points where I differ. Have a good day.
graemep 1 days ago [-]
The internet pre-AI did far more to make knowledge accessible than AI has done. Its useful, but its not the critical thing.
maxglute 1 days ago [-]
I want fully automated luxury communism... aka ai + abundance. Obviously I dont want to go through the profound dislocations and transformations that gets us there, especially not if I won't live to see it. But some people sometime will be better off with AI... and then extremely worse off once ai figures out how to remove mouths.
hoppp 1 days ago [-]
I think we did good inventing it and it's inevitable. Lets use it well.
1 days ago [-]
hisfraudulency 1 days ago [-]
Why are the using the Ukrainian transliteration of Chernobyl now
chuckadams 1 days ago [-]
Because it’s in Ukraine? Same reason we spell the capital Kyiv now.
tpoacher 1 days ago [-]
Indeed, I spotted that too and thought it was interesting. Biblical almost.
baggachipz 1 days ago [-]
> Hinton, 78, said that within a decade and possibly sooner, humanity will know whether it can manage the technology’s risks and enable what he called “radical abundance” or whether “we fail to manage the risks and … have a huge gap between rich and poor, huge cultural problems, massive unemployment, just society being torn apart”.
Uh... we're already at the latter prediction.
maxwellito 1 days ago [-]
I'm afraid I can't do that Dave.
bryanrasmussen 1 days ago [-]
almost every story that has ever been written about a world with AI has posited that humanity was better off without AI.
mrweasel 1 days ago [-]
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money. These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
mirmor23 1 days ago [-]
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
Aurornis 1 days ago [-]
This is an interesting theme that I see a lot: Some people are firmly anchored to an idea of AI capabilities that is a year or more behind the frontier.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
mrweasel 1 days ago [-]
> but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
That wasn't the point. My point was that if you had rolled LLMs out like that, as embedded technology in existing products. Slowly, over time, you'd have a lot less cynical people. I think the hostility towards AI product in large parts comes from the disruption and glancing over the negative parts. What we're doing now is the equivalent of putting millions of cars on roads which five years ago had never seem a car, only horses. We're doing so without any understanding of how that might affect society. Social media was a similar story. It was slower, yet we still didn't catch the majority of the negative consequences.
If it turns out that the people behind social media, and smart phones, don't care about e.g. peoples mental well being, then why would I trust that the people behind AI wouldn't behave the same way?
cryptopian 1 days ago [-]
Cars are actually a great historical example of this. The car was invented, and people saw the benefits, but some parts of the world went so car-mad that they started bulldozing entire neighbourhoods and deprioritising anything that wasn't a car. Decades later, you see the knock-on effects of those decisions, with high traffic levels, pollution, people spending hours in cars because it's the only way to go shopping, see friends, or take the kids to school. Places that took a more cautious approach still gained the benefit of cars, but kept the choice and quality of life for others
rhdunn 1 days ago [-]
As I see it there are currently 3 broad classes of LLM usage:
1. enhanced autocomplete -- either line (e.g. JetBrains small language models) or longer predictions;
2. chat -- asking questions related to the task you are doing;
3. tool assisted/agentic -- letting models interact with tools to perform tasks.
I personally find 1 & 2 helpful for my development tasks. For 3, I'm cautious of using it for several reasons:
1. lack of control of what gets sent to the model (e.g. if you have confidential user data, proprietary code and/or data, access tokens/private keys);
2. letting the models run arbitrary commands (modifying the system, deleting files, etc.);
3. not having control over the design process, formatting of the files, etc.
1 days ago [-]
knottn 1 days ago [-]
There are people whose livelihood depends on them not getting it. There are people whose worldviews depend on them not getting it. Denial is a part of grieving. There are also people who like to troll the ai maniacs because ai maniacs are easy to troll. It has been curious to me to see all this as well, smart people who seem oblivious feels strange.
Telemakhos 1 days ago [-]
AI has an extremely high capital barrier to research and development. Getting that initial capital required promises of a world-changing technology. A few companies with lots of investment were able to develop the technology and roll their own models. The business case for this R&D was that providing the models to paying users would amortize the R&D costs. But, promising disruption was the first step to getting the money.
Then the Chinese came along and showed that you can take those model weights, repackage them as open weights, and undercut the business model. Suddenly Dario et al. start talking about how dangerous AI is and how it needs to be regulated right now, before someone's abliterated open-weight Chinese-provided business-undercutting model destroys the world. They want their walled gardens regulated into being the only game available.
Both stages of the game—raising capital and regulatory capture—require hyping up the disruptive aspect of AI.
dboreham 1 days ago [-]
Insiders were warning of dangers years before Chinese models.
senordevnyc 1 days ago [-]
Didn't Dario leave OpenAI to start Anthropic specifically because he was worried about the risks and didn't think OpenAI was taking them seriously enough?
tim333 1 days ago [-]
> Had LLMs first been presented as a "better IntelliSense"... very few people would have been upset.
That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not just about LLMs any more than the industrial revolution was just about steam engines.
roenxi 1 days ago [-]
There are a huge number of people involved in AI outside of the major AI companies both in smaller AI companies and lots of independent developers. They aren't having much success bringing AI to the masses and this is because they don't care enough about money and they aren't selling their ideas aggressively enough. There is this odd tendency to undersell achievements because they weren't motivated by the Right Stuff.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
ontouchstart 1 days ago [-]
What would be the Right Stuff?
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
geraneum 1 days ago [-]
> There are a huge number of people involved in AI outside of the major AI companies
How many, approximately?
mesofile 1 days ago [-]
The thing I keep thinking about with this sort of behavior is how Zoom became big in video conferencing. They had good features, yes, and COVID obviously gave every player in that space a big opportunity, but Zoom distinguished themselves in great part by doing every underhanded shady thing they could to make their app 'frictionless', privacy-invading, data-grabbing, and generally preyed on end users' lack of awareness. They did not keep doing all of those things, as far as I know – they got called out a lot but I have steered clear of them since my initial experience with them so I don't know how much they cleaned up their act. But taking the most aggressive and least ethical approach certainly seemed to have helped them get to the top at a very competitive time, and once there they could probably afford to stop doing the most egregious things.
It seems obvious that this same dynamic is in play with the frontier AI companies now. They know if they win the race to the they'll be forgiven their sins, or anyway they'll be beyond the reach of effective punishment.
> Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations
Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.
It's sorta crazy to think how many years ago that was now.
pier25 1 days ago [-]
> These are also people who aren't accustom to cleaning up their own mess, so what do they care if something goes wrong
They certainly don't care when a model hallucinates or an agent does something it shouldn't be doing. It's your fault and they charge you for the tokens.
highfrequency 1 days ago [-]
It’s more than that. Scaling LLMs had a fundamental problem - that it took $10b+ of investment to become useful. To get over this huge activation hump, you had to hype the thing as world changing. If you sold it as autocomplete you would never raise the money to do even autocomplete.
That’s why the AI heads all have extremely dramatic narratives. They are the end product of a selection tournament for raising massive amounts of money pre-product.
nater5000 1 days ago [-]
>Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset.
You make it sound like this was a feasible option. This technology would exist regardless of how any one company decides to frame it. If OpenAI decided to go with a more conservative approach rather than just throwing ChatGPT out for the masses, then all it would take is for one person to realize they didn't take full advantage of this technology to just make their own ChatGPT.
>The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
You can criticize the current tech leaders all you want, but what you're describing is a fundamental dynamic of capitalism. You must choose the most profitable option, otherwise you have chosen the wrong option. If the current tech billionaires all vanished just as ChatGPT was released, they'd be quickly and easily replaced with roughly equivalent individuals who would effectively operate the same exact way. That's true because they're operating based on the mostly deterministic policy of profit maximization. They don't really get a say in it; they must follow the path towards maximizing expected future returns, otherwise they'll get canned by the investors who are seeking maximized expected future returns.
None of this relieves them of their personal ethical responsibilities either. I'm just pointing out that your comment might as well be describing an alternate reality where everyone lives peacefully with each other and nobody goes without what society can afford to offer them. It's equally as unrealistic.
Aurornis 1 days ago [-]
> The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
rafterydj 1 days ago [-]
I really don't like the excuses that are inherent to the "well it was inevitable" line of thinking. Oppenheimer was still responsible for the bomb despite it being "inevitable" in a similar way.
Despite this, when discussing the egregious behavior of the frontier labs, there is vanishingly little acceptance of the idea that 1) yes, it probably was inevitable in some ways AND 2) those involved, particularly at the top, are still responsible for the consequences of their (bad) decisions.
mcv 1 days ago [-]
Both nuclear bombs and AI are not inevitable. Technology depends not just on knowledge, but on infrastructure. Building and maintaining nuclear bombs requires enormous amounts of resources, and it's a conscious decision to continue spending those resources to maintain nuclear capability. Similarly, developing AI requires enormous resources, as you can see from the massive investments in hardware and data centers. If the economic foundation for it doesn't exist, you're not going to get more powerful AI.
The models we currently have aren't going away, and especially the ones small enough to run at home will continue to be used, but if these data center investments end up bankrupting Big Tech, I don't think we're ever going to see anything beyond that. Not for a long time, at least.
e28eta 1 days ago [-]
Does developing AI _require_ enormous resources, or is it using enormous resources right now because the people with the capital are throwing it at the AI companies hoping for a massive payout in the future?
My guess would be that once the money runs out, things slow down, but incremental progress continues to be made. A focus on smaller models will improve their capabilities. We’ll get better at driving them. It’ll be used in places where it truly makes sense, instead of anywhere & everywhere just because it’s easy.
I don’t actually know what the financials are, but even without the effective subsidy from VC money, I have to believe there are things AI’s doing today that’s way cheaper than what it would cost to have a human doing equivalent work. And that dynamic will drive further innovation in the space for a long time.
munksbeer 17 hours ago [-]
> Both nuclear bombs and AI are not inevitable.
What does this actually mean? In what sense are people using this term. In the most strictly literal sense then sure, nothing is inevitable, because all paths are open.
But in the practical sense, I am quite happy to use the term inevitable for many technologies. I think if ASI is possible (and I think it is), then it is also inevitable. I would happily work towards developing ASI technology, because I believe it will be ultimately good for the world. But it sounds like a lot of people want to lock me up or even kill me if I did that.
pyyxbkshed 1 days ago [-]
I have also not been able to lean towards a side in inevitable v/s responsible.
1. Is it both inevitable and still they are responsible?
2. Or it was inevitable and they were just an instrument to make it happen.
3. Or it wasn't inevitable and if it were not for some specific people then it would not have happened?
I am leaning more towards 2. But if that is the case why are only specific people reaping the benefits of this?
If it is 3 or 1, then why aren't we holding them accountable?
watwut 1 days ago [-]
"Inevitable" means "same technological progress would be achieved anyway, even if these people have made less damaging choices".
So yes, they are 100% responsible for damage they caused and did not had to happen. And no, "I had no choice, because it was the only way to become billionaire" is not an excuse.
They were NOT "instrument". No one manipulated them. No one forced them. Nothing forced them. They themselves wanted to and did not cared
mchusma 1 days ago [-]
I am not sure what gripe you have with the labs, but I think if you are worried about AI then you got shockingly lucky with the lab leadership. My biggest gripe has been the campaign of negativity against ai.
jfyi 1 days ago [-]
Sorry, yeah I don't trust Altman and Musk, the companies they run inherit that. Their public face betrays an almost cartoonish evil, it's sad people can't see that.
Other labs seem to be working the exact same angles as them, but just have the good sense to not be so open about it. It's not the "lab" people have a problem with, it's their leadership. These people cannot be trusted to make the right decisions for society with this technology.
WarmWash 1 days ago [-]
>campaign of negativity against ai.
People are starting to panic, and few things bring out full blown idiocy and disregard for ground truth faster than widespread panic.
goatlover 1 days ago [-]
It's hard not to be when people at those labs talk about the probability of AI killing us all, and if it doesn't do that, then the massive disruption to the labor market that's coming. Combine that with the techno feudalist ideology of leadership.
So yeah, I'm negative against their development of AI along with the general public, and for good reason it would seem.
Also, I don't want super intelligence. I want humans to remain in control. Why are they trying to build something to replace humanity? Why don't we got a vote in this? All 8 billion of us not developing frontier models with the expressed goal of achieving AGI followed closely by ASI.
pelotron 1 days ago [-]
We didn't get a say in whether we should bet our entire economy on building out infrastructure for half a dozen guys to get rich from. No say in the way they are backdooring around the hamstrung EPA to install greenhouse gas machines to power these things. I think the pushback makes sense.
It's a pity because as computer scientists this should have been such a fun technological moment for us. What a cool thing to experiment and explore new open doors with. But instead it was immediately the driver of mass job loss, societal upheaval, and a double digit percent chance of ending humanity, espoused by the AI lords themselves.
munksbeer 17 hours ago [-]
> Also, I don't want super intelligence.
I do. Quite a lot of people do. And that is because we believe it will lead to a better life for most humans.
goatlover 2 hours ago [-]
I don't think you're anywhere close to a voting majority. Rather quite a small minority of the human population would want such a thing is my guess. It may seem otherwise in your bubble though.
hardbass 1 days ago [-]
Why do you want to block people who are open to the idea of becoming transhuman/hybrids? Let it play out and see whose society turns out better.
goatlover 2 hours ago [-]
Do that without dragging the rest of us into it then. I will continue to vote to block and advocate against such a thing on a societal level. It should not be thrust upon humanity without our consent.
bluefirebrand 1 days ago [-]
> Why are they trying to build something to replace humanity?
The list of possible reasons is pretty short and mostly not really good for most of us, as far as I see it
bigstrat2003 1 days ago [-]
> if you are worried about AI then you got shockingly lucky with the lab leadership.
You must be joking. Altman is a sociopath, Amodei is either too stupid or too evil to stop working on tech he claims will have a 10% chance of destroying humanity (and in either case he should be imprisoned to protect the rest of us from him), and Musk is a man-child who will lash out at anyone and everything who dares to speak ill of him. There's absolutely nothing "lucky" about having these jokers in charge of stuff.
hardbass 1 days ago [-]
Other than Musk, the other two seem to me overall quite good stewards. Could have been much worse, and indeed hard to find someone better. How do you think someone could have been a better steward? There are 8 billion people in the world, almost all of them believe in absurd and mutually incompatible imaginary tales, its a very hard thing to create an information agent that doesn't offend a big fraction of the world.
bigstrat2003 7 hours ago [-]
I think it's pretty trivial to imagine ways in which they could've done better. For example, if you genuinely believe some tech has a 10% chance to destroy everyone alive, stop fucking developing it. There is no scenario where it is remotely acceptable to pursue such a risky goal. Amodei is either too stupid to see that, in which case he can't be trusted with anything sharper than a butter knife, or he's too evil to care about the risk, in which case he can't be trusted with anything at all. Or, more likely, he's lying to hype up his company, in which case he's a fraudster.
It's similarly easy to come up with "what Altman should do better". The dude is a genuine sociopath in charge of a company trying to make everyone's lives worse off in order to enrich him and his investors. It doesn't get much worse behaved than that.
hardbass 6 hours ago [-]
If you impute its bog standard marketing/hyping, its the least evil and most mundane possible things you can expect in owners of such powerful tech. Your imagination is weak. Imagine if the owner was a Christian nationalist or Muslim fundamentalist as some examples.
eidieiej3838 1 days ago [-]
The Oppenheimer comparison is very apt I think. Even if the science was inevitable, the decision to use the output (twice) in arguably unnecessary circumstances was not
ThrowawayR2 1 days ago [-]
The other difference is that Oppenheimer at least felt terror and remorse. The AI bros are saying "what we built might end the world but, lol, whatever."
1 days ago [-]
knottn 1 days ago [-]
“not acknowledging the inevitability of the technology being developed further”
This looks a lot like the “if I didn’t do $badthing someone else would have” line of reasoning.
bensyverson 1 days ago [-]
Just to flip it around on you, what do you think would happen if Anthropic and OpenAI agreed to stop working on AI altogether?
I'm not arguing about the consequences or responsibility. I'm asking which humans would be capable of stopping the development of AI at this point in history.
Aurornis 1 days ago [-]
The comments on this are interesting. There are a lot of assumptions from different people speaking about humanity as a single entity that decides how to operate as one. A lot of others act as though it’s really easy to get world governments to all agree to not develop a technology through treaties and agreements. One person below is arguing with me by saying that we “just” need to get China and the US to agree to halt AI and the problem is solved for the whole world.
It’s difficult to understand where this all is coming from because it’s so strangely disconnected from history. It’s not even consistent with current world events. The US is in a war with Iran over nuclear proliferation while also worrying about North Korea’s nuclear program. China and the US are in a trade war and Taiwan is under threat due to semiconductor capabilities (among other things, obviously). Open weight models allow anyone in any country to do AI work on their laptop. The top story on HN yesterday was a small team of academics making some cool discoveries on improving LLMs without frontier lab level compute.
How anyone can see all of this, in addition to decades of history of attempted nuclear diplomacy and wars, and posit that the whole problem is easily solved by simple agreements is confusing to me. The confidence with which these alternatives are being proposed and the unwillingness to acknowledge second order effects (like AI researchers moving to a different country and continuing to work) is baffling.
hardbass 1 days ago [-]
8 billion people. Almost all of them aside from a negligible fraction believe in at a fundamental level completely mutually incompatible fairy tales, a single digit or so fraction of them are killing each other over these tales. And some people think the world over would simply get on one page over AI?
bensyverson 1 days ago [-]
hn is full of technical people that like to believe that they behave logically (they don't), and that others would change their behavior if the logic was explained to them (they won't).
But yeah, there are a lot of comments from genie-rebottlers that are not very grounded in reality. They're functionally equivalent to saying "hey, everyone in the world, let's not actually use electricity after all" in 1905.
WarmWash 1 days ago [-]
People don't understand the prisoners dilemma, or they think it is some kind of psychological pseudo science trumped up to make fun pop-sci articles for social media.
reasonableklout 1 days ago [-]
But building AI isn't the same as defecting in the Prisoner's Dilemma because it risks harming the defector too, if there is loss of control.
If we're talking game theory, a better model is the Stag Hunt [1]. At some point the equilibrium strategy is to cooperate.
Another difference is that real life is an iterated game, and the iterated prisoner's dilemma rewards cooperation.
Maybe AI lab CEOs will change their behavior once they read about it /s
WarmWash 1 days ago [-]
Probably not, because the are already acting in accordance with it.
orsorna 1 days ago [-]
This was the logic for developing the atomic bomb and it was a good call.
josephg 1 days ago [-]
Was it? We nearly destroyed ourselves with nuclear weapons several times in the last century. I don't think it was a good call just because we've gotten lucky so far.
WarmWash 1 days ago [-]
Better than someone else actually killing us all because they went at it totally unrestrained.
If there is a room full of people, and one person gets a gun, who would you pick to have it?
Yourself, obviously.
josephg 1 days ago [-]
Good thing then that America's invention of nuclear weapons kept nuclear weapons out of the hands of the russians. Oh wait.
Maybe america will end up the only world power holding the secrets of LLMs? ... Oh no...
WarmWash 1 days ago [-]
There were no secrets of nuclear weapons. The gating factor is the ability to purify uranium.
If nuclear weapons were as easy to build as AI models are to run, we would have all been dead decades ago.
josephg 1 days ago [-]
> There were no secrets of nuclear weapons.
There might have been. But the notes from the Manhattan project were leaked almost immediately to the russians by informants.
HarHarVeryFunny 1 days ago [-]
Sort of, although didn't Von Neumann package it slightly differently - less personal (someone else) and more about the evolution of science and separation of roles of scientist and politician... basically science is going to happen, and it's not for people like us (scientists) to be worry about what happens as a result.
Still seems a moral cop-out, and while true, the timeline would certainly have changed, if not the inevitability, if key figures such as himself had opted out.
1 days ago [-]
mcv 1 days ago [-]
After the war, it turned out that the Nazis weren't even working on a nuclear bomb. So had the US not done it, nobody might have. Other nuclear powers did it only because the US had them and had shown that it was possible.
> Nazi Germany undertook several research programs relating to nuclear technology, including nuclear weapons and nuclear reactors, before and during World War II. [...] The first effort started in April 1939, just months after the discovery of nuclear fission in Berlin in December 1938, but ended shortly ahead of the September 1939 German invasion of Poland, for which many German physicists were drafted into the Wehrmacht. A second effort under the administrative purview of the Wehrmacht's Heereswaffenamt began on September 1, 1939, the day of the invasion of Poland. The program eventually expanded into three main efforts: Uranmaschine (nuclear reactor) development, uranium and heavy water production, and uranium isotope separation. Eventually, the German military determined that nuclear fission would not contribute significantly to the war, and in January 1942 the Heereswaffenamt turned the program over to the Reich Research Council (Reichsforschungsrat) while continuing to fund the activity.
> The program was split up among nine major institutes where the directors dominated research and set their own objectives. Subsequently, the number of scientists working on applied nuclear fission began to diminish as many researchers applied their talents to more pressing wartime demands.
> [...] Heisenberg said that in 1939, the physicists at this second meeting concluded that "in principle atomic bombs could be made, but they also emphasized that it would take a number of years",[24] certainly not less than five. He also said that he "didn't report it to the Führer until two weeks later and very casually because I did not want the Führer to get so interested that he would order great efforts immediately to make the atomic bomb. Speer felt it was better that the whole thing should be dropped and the Führer also reacted that way."
> [...] Speer states that the project to develop the atom bomb was scuttled in the autumn of 1942. Though the scientific solution was there, it would have taken all of Germany's production resources to produce a bomb, and then no sooner than 1947.[39] Development did continue with a "uranium motor" for the navy and development of a German cyclotron. However, by the summer of 1943, Speer released the remaining 1200 metric tons of uranium stock for the production of solid-core ammunition.
There are ways of calling this "weren't working on a nuclear bomb", but it seems to me like "they wanted to, were moving in that direction, and would have done it if they had the resources and weren't distracted by shorter-term considerations".
mcv 24 hours ago [-]
> the project to develop the atom bomb was scuttled in the autumn of 1942
That sounds very much like "they weren't really working on it". They explored the options and decided not to.
Whatever you want to call it, that's what I was referring to.
goatlover 1 days ago [-]
The development lead to it being dropped on two cities followed by an arms race that put global civilization at permanent risk, with several incidents already that could have launched a nuclear war between superpowers.
How is any of that good?
pixl97 1 days ago [-]
It's likely stopped lots of large direct wars between the biggest nation states. Instead it's much smaller proxy wars.
---
Simply put humans are at risk of any power scaling. We exist safely in a very tiny temperature and energy environment and the rest of the processes in the universe operate widely out of that strict safe area.
It's also a catch-22. There are large rocks in space that will eventually end us unless we do something about that. But having the power to do something about that gives us the same power to also end everyone on the earth.
Aurornis 1 days ago [-]
Academic research has always progressed and pushed the boundaries. Is that a $badthing to you?
josephg 1 days ago [-]
Imagine a bag filled with marbles. We can't see inside the bag, we can just pull marbles out of the bag. Some marbles are white - they improve the world in some way. Wifi. Trains. Electricity. Some marbles are grey - they are useful but dangerous. Firearms. Nuclear weapons. Gene editing. And some marbles are black. If we pull a black marble out of the bag, we hurt or destroy ourselves.
Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]
And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.
> Humanity's strategy so far has been to pull marbles out of the bag as fast as we can.
> But why are we rushing so much?
Talking about these things with elaborate analogies only obscuring the point you already stumbled into: Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.
Talking about humanity like one entity that operates together misses the point: There are billions of people in the world and some of them are going to discover new things.
There is no general counsel where humanity gets together and decides what humanity works on. There is no singular bag with a singular “humanity” pulling marbles out one at a time.
The analogy is too broken to be useful for the discussion.
haswell 1 days ago [-]
There’s a point at which logic/reason/wisdom overrides our curiosity, and there are quite literally multi-nation groups that collaborate on matters of global importance.
There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.
Aurornis 1 days ago [-]
> There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.
You’re speaking of “we” as if the whole world was operating in unison, while there’s a war going on in Iran over nuclear disagreements.
The nuclear situation only reached stability after multiple countries fully developed it. I don’t see the parallels to halting AI.
haswell 1 days ago [-]
It's hard for me to understand how the parallels wouldn't be abundantly clear.
The nuclear stabilization that occurred was the result of a collective understanding of the existential risk it posed leading to anti-proliferation agreements and superpowers going to great lengths to prevent nations deemed as unstable from obtaining such technology.
Some portion of the global community may collectively discover/decide that the risk AI presents is worthy of anti-proliferation treaties and wars may be fought over it.
The main gap in the comparison is that we have a much clearer understanding of the risks of unchecked nuclear weapons use than we do about what current and future AI tech might or might not unleash.
The underlying point is that we have historically placed limits on our curiosity when we deem certain paths of exploration existentially dangerous. It has yet to be seen if AI will reach those levels of existential risk, but if they do, there is plenty of precedent for stepping on the brakes.
josephg 1 days ago [-]
> There is no general counsel where humanity gets together and decides what humanity works on.
Nuclear Non-Proliferation Treaty (1968)
Biological Weapons Convention (1972)
Asilomar Conference on Recombinant DNA (1975)
We've done this sort of thing plenty of times around specific, dangerous technologies.
pixl97 1 days ago [-]
And countries still violate these treaties. See Iran and NK.
Also the bio/DNA ones are quickly falling apart as the technology to do these kinds of things on a small scale are readily available. To the point where we're finding illegal wet labs in the US.
Nuclear was more of an exception because of many of difficulties in purifying stuff for weapons and its detectability.
AI is a much, much, much harder problem is it uses the already existing infrastructure as a base. Unless we regulate, monitor, and ban all private usage of technology AI can be developed at a small scale.
josephg 1 days ago [-]
Is your argument essentially that because global treaties haven't been universally effective in all cases, we should just YOLO AI even if it has the potential to cause great harm?
pixl97 20 hours ago [-]
No, we shouldn't yolo AI, but because we started the ball rolling and it may be impossible to stop.
Planktonne 1 days ago [-]
> Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.
This is true, but not all the time, and transparently not the case for the current leaders in AI development. Their motivator is money.
hardbass 1 days ago [-]
Whats wrong with gene editing?
Loquebantur 1 days ago [-]
It's not about merely knowing how to build a "badthing". It's about knowing why not to.
With AI, the point is who uses it for what.
Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?
miltonlost 1 days ago [-]
And this sort of blanket argument is how you get bad science on the order of Prison experiments to Nazi scientists. Yes, academic research that pushes boundaries can be a $badthing
asdaqopqkq 1 days ago [-]
we can outlaw those kinds of uses, im kinda split rn, most people arent happier with AI but it is empowering so many people, especially around obtuse labor laws etc, it is a double edged sword squared
Loquebantur 1 days ago [-]
You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.
Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.
The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.
With AI, that reality gap will swallow whatever is left.
hogs_get_fat 1 days ago [-]
The market is irrelevant. In the 28 centuries of recorded history, the human race has pursued power in it's various forms (political, military or economic), without regard to the "public good" (which is really, just a polite euphemism for the established order and those that benefit for it).
Without capitalism, you wouldn't be able to restrain AI adoption any more than the armored knights of the middle ages were able to restrain the adoption of the firearm.
Aurornis 1 days ago [-]
> You operate within the "logic" of capitalism, where no other value exists but the (short term) economic one.
I operate within the logic of how the world actually works.
Which I find infinitely more useful for practical topics like the reality of AI development in a global economy.
Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
nananana9 1 days ago [-]
You're acting like international treaties (or cartels) don't exist. Yes, these are structures where parties have incentives to defect, and we counter those by creating incentives not to, with checks and sanctions.
Additionally, you don't need all the leaders of the world to agree to anything, you only need US and China.
Additionally x2, if you agree with the common stance that China makes the vast majority of their progress by distilling US models, you don't even need them to agree to anything.
pixl97 1 days ago [-]
A ban on AI looks nothing like a ban on almost every other technology that has come before it.
You're really suggesting the following bans. A ban on computer science and algorithms. A ban on any device that can play games that aren't encrypted or DRM locked from an approval authority. A ban on any kind of unregulated computing without government approval and licensing.
>China makes the vast majority of their progress by distilling US models
Eh, this is a budgetary decision, not a technological one. Why spend billions when the US can spend billions and you can spend millions. China more than has the power to do what we're doing if they chose to do so, their biggest limitation is speed of making new compute, not money or brains.
nananana9 15 hours ago [-]
> You're really suggesting the following bans. A ban on computer science and algorithms. A ban on any device that can play games that aren't encrypted or DRM locked from an approval authority. A ban on any kind of unregulated computing without government approval and licensing.
You know what, fuck it, that's exactly what I'm suggesting. Let's ban personal computers. The only consumer hardware allowed should be a HTTPS client directly burned on the chip that can only connect to government servers. Absolute all timer of a straw man. This is probably the last comment I'm writing on this site, this place has become unsufferable.
Believe it or not, there already are things you're not allowed to do with computers that you otherwise own, and we have ways of enforcing them other than trusted computing -- the threat of putting you in a cell for a few decades usually does the job. That's without getting into the fact that you're not going to train 1T-sized models on your laptop, and billion dollar GPU farms aren't precisely hard to keep track of.
sznio 1 days ago [-]
>Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
it has kinda worked for nuclear weapons
defen 1 days ago [-]
"We" (Israel, probably with US assistance) assassinate "rogue" (outside of existing nuclear powers) nuclear scientists (e.g. from Iran). So to take this analogy to its conclusion you have to accept that part of the pact would be assassinating people for doing computer science research or building computer systems that will be used for AI.
lirolero 1 days ago [-]
[dead]
WarmWash 1 days ago [-]
Nuclear weapons come with the immense benefit that purifying uranium is so challenging, expensive, and difficult to hide that we don't even bother to protect the information on how to build a nuke. They also have a second benefit of being impossible to test without the whole world knowing you did.
AI has no such hurdles.
reasonableklout 1 days ago [-]
No, the AI compute supply chain is actually quite similar to the nuclear supply chain. Building chips capable of training or large-scale inference is extraordinarily complex, expensive, has a large footprint, and has a few chokepoints (rare earths mining, HBM manufacturing, and ultraviolet lithography, where there is literally one company in the world capable of building the relevant outputs) where we in fact already have governance mechanisms for things like export controls.
Aurornis 1 days ago [-]
In the nuclear case multiple powers raced to develop it as fast as possible and as much as possible.
That’s the inverse of what I said.
Loquebantur 1 days ago [-]
The "world" isn't just what humans currently make of it. You deny the possibility of alternatives for society based on what amounts to ignorance.
Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".
When you imagine people as savages to be outdone in their savagery, you become one yourself. Spend more effort on finding ways not to.
Aurornis 1 days ago [-]
> The "world" isn't just what humans currently make of it. You deny the possibility of alternatives for society based on what amounts to ignorance.
Ignorance of what? All of the previous successful attempts where the entire world reorganized itself into a single world government?
> Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".
I think you’re putting the cart before the horse. You’re assuming it’s necessary and then making another leap to assume that being necessary means it must also be possible.
ford 1 days ago [-]
Capitalism isn't the most palatable economic system, it's the one that best matches the reality of nature (competition)
I think the vast majority of the rich are greedy bastards, but IMO being rich does not imply greedy bastard 100% of the time.
goatlover 1 days ago [-]
Arguably humans cooperate even more than they compete. Examples of cooperation are easy to find throughout nature. So is parasitism, but humans generally find that loathsome as a human activity.
The argument from nature is a fallacy. Just because something is doesn't mean it ought to be.
hardbass 1 days ago [-]
Humans also love killing each other enmasse, especially by claiming someone or some group is destroying the good cooperative group.
gilrain 1 days ago [-]
> it's the one that best matches the reality of nature (competition)
This doesn't seem right: it doesn't match the history of reality, in which capitalism has repeatedly stymied rather than fostered competition.
wombat-man 1 days ago [-]
better intellisense is how it started for me at least.
sneak 1 days ago [-]
The AI formula is out of the bag. The Manhattan Project is over, trinity has happened.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
233mhz 1 days ago [-]
> If it weren’t US companies, it would be others.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
andsoitis 1 days ago [-]
> There is something deeply rotten about the US mindset
What is that “something”?
233mhz 1 days ago [-]
Profit above all else, deregulation at all cost, "we're the police of the world", christian brained politicians, etc.
andsoitis 1 days ago [-]
> profit above all else
The world’s biggest philanthropists are American.
The US has for decades invested into charities and human development around the worlds .
> deregulation at all cost
The US has regulations. The Constitution of the United States is the oldest and longest-standing written and codified national constitution in force in the world.
> Christian brained politicians
We have separation of church and state.
JSR_FDED 1 days ago [-]
It’s precisely because the corporate mindset of “profit before all else” that certain individuals become fabulously wealthy, enabling their philanthropy.
The Sackler family of opioid infamy were huge philanthropists. Not saying that the AI companies are like Purdue Pharma, but clearly philanthropy does not compensate for societal damage.
andsoitis 1 days ago [-]
Cognitive dissonance. How can you say with a straight face “profit above all else” and then also concede the huge philanthropy?
Americans and the US as a whole are the most charitable in the world.
The data does not lie.
jfyi 1 days ago [-]
It's only cognitive dissonance if you look at the "US as a whole".
Americans can be charitable individually, but that does not contradict the reality that US corporate and institutional structures operate on a strict "profit above all else" mandate.
> Americans and the US as a whole are the most charitable in the world.
Yeah because it includes lobbies and turbo billionaire pastors from texas using "charity" money to buy rolls royces as well as other highly retarded """charities""" that aren't even legal in the rest of the developed world
233mhz 1 days ago [-]
> We have separation of church and state.
You have senators and wannabe senators quoting the bible on TV as their guiding star as we speak right now, give me a break
On top of that you can separate the church and the state all you want, when people like Thiel show up it doesn't matter, these people are deeply troubled, if they were muslim and saying 1/10th of what he says you'd send them to guantanamo
hardbass 1 days ago [-]
Better than in the hands of a rabbi or ayatollah or christian extremist or various others. I feel simply profit minded people are one of the less dangerous hands in which powerful tools can exist.
carlosjobim 1 days ago [-]
I remember reading about the development of neural networks in a magazine from the 1980s.
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
hattmall 1 days ago [-]
How can "AI" facilitate genocide? Like what might chatgpt have suggested that would have made the nazis more effective or the khmer rouge.
ks2048 1 days ago [-]
You really can't think of anything? How about giving a fleet of AI drones a large supply of bullets and telling them to go kill everybody in some region or of some group.
sneak 1 days ago [-]
Widespread mass realtime facial recognition usage is allowing governments to target and repress subsets of the population with an unprecedented scale and efficiency. ICE are flying facial recognition drones over sidewalks in downtown LA. Dissdents are having their facial geometries entered into secret databases simply based on political opinions.
carlosjobim 1 days ago [-]
Any new technology throughout history has first been used for war and genocide.
How could modern AI have helped those groups you mention in their endeavour? Really fucking easy and effectively, and you don't need a lot of imagination to think of how. From surveillance and interrogation to pinpoint determination of ethnicity and much more. Not to mention translation between human languages.
233mhz 1 days ago [-]
> the nazis more effective or the khmer rouge.
They literally targeted people for their past opinions, how they looked, etc. which you can now determine and process at break neck speed, you can even remove humans from the loop and have the machine determine who's an undesirable or not.
How is this a serious question lol?
233mhz 1 days ago [-]
> Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did
Uh? In what universe are openai and anthropic not shadow groups?
carlosjobim 1 days ago [-]
Are you misunderstanding on purpose? OpenAI and Anthropic let the public use their AI models. Shadow groups would mean that people wouldn't even know about the existence of AI.
233mhz 1 days ago [-]
> OpenAI and Anthropic let the public use their AI models.
Oh, how nice of them, such charitable souls...
hardbass 1 days ago [-]
Do you take a salary or ask money for your products?
1 days ago [-]
customguy 1 days ago [-]
> If it weren’t OAI and Anthropic, it would be others.
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
hogs_get_fat 1 days ago [-]
Rushing towards creating machines that can give you vast amounts of military, political or economic power without regard to it's effects on the old order.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
customguy 1 days ago [-]
Well, are you thinking nation states? Then you kinda also need a population that is cool with a sort of blitzkrieg being waged "on their behalf" against the rest of the world (and they should think twice if that machinery wouldn't also logically end up being turned onto them). Otherwise, what's the race for, unless to use it offensively? So you either subjugate or eradicate the world, and then what? Who wants that?
Political power is when people act in concert, when leaders have people behind them because those people agree with how they're being lead etc., it's not something that happens unilaterally. You can physically force people, but for what when you can have robots that you don't have to force?
So that kind of leaves wiping out nearly all humans, for a few hundred thousand remaining people to get served by robots and play golf. Which I would say is what we're kinda on track for. But that wouldn't be a race between nations, that race would just be pretend to get the tech online; it would rather be a race between all humans and a bunch of sick people.
hogs_get_fat 1 days ago [-]
I'm thinking of nation states, but I fear you have a very limited view of how power is created and yielded.
AI doesn't have to be used offensively to give a nation power. The machines can already design new software at the drop of the hat, it will at some point become capable of designing new consumer and industrial products. Putting a tireless intelligence at the task of looking for new optimizations in your industrial processes and factories will eventually give you economic dominance, one way or another.
Offensive uses don't need to be as extreme as you imagine them. We're already starting to see the effects of having meticulous automated hackers. Instead of a blitzkrieg, imagine your foes suffer from a constant stream of industrial accidents, unfortunate events and information leaks that can never quite be traced to your nation.
Finally, political power itself doesn't need to come from people acting in concert. There's always been wheeling, dealing, intrigue and bureaucracy. Once AI is sufficiently versed in legal matters, it'd allow political interest groups to overwhelm governments and private enterprises with objections and appeals, paralyzing their ability to carry out their agenda until they make concessions.
The possibilities of using intelligence for acquiring more power are literally endless..
HarHarVeryFunny 1 days ago [-]
Yes, the cat is out of the bag as to how to create this type of AI.
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
aatd86 1 days ago [-]
Which job are worth doing and which ones you'd rather the AI do?
AI should enable things that were previously impossible because of time constraints.
If time is money and we were previously trading with money, we will get back to trading time. Manufacturing time for instance. Compute times.
For commodities, that might be automated to some extent, I am not sure. The friction in exchanges is where the money is made and the friction here may be at the interface with the real world still.
hardbass 1 days ago [-]
At some stage we might need to decouple income from traditional jobs. Bringing gladiatorial combat would be helpful. People can't stand a life of comfort. It will both probably become attractive to watch and a source of living for all, and it will solve the devils that emerge in idle heads. I am not joking, just look at the state of the Western world just from some basic material comforts having been fulfilled for a few decades.
rancar2 1 days ago [-]
I think this recent post about a 2017 AI safety risk review may give some solid historical context that many of the leaders do not want to be the next Oppenheimer and have taken steps overtime to prevent being in that role themselves personally and collaborating with others so no one is: https://news.ycombinator.com/item?id=50366387
The only leader I’m not aware of really taking a more active role in safety on the daily is Elon Musk, so I’d like to see evidence supporting what he is doing or not doing on that front if someone from SpaceX can share their experience. The evidence on the environmental front from SpaceX is very concerning and is disregarding the wellbeing of the world and local populations in my opinion.
herculity275 1 days ago [-]
Musk has recently shifted to openly advocating for full-speed accelarationism, claiming that being a "biological bootloader for AGI" is a success condition for humanity. He's pretty clearly not concerned with preserving anything about the current state of the world.
hardbass 1 days ago [-]
Not against that concept in theory if it managed to be executed properly. Gene editing and centaurism always seemed natural next steps to me. I am willing to treat the viewpoint of some people wishing to slowing down or temporarily stopping such work in the sense that we must do a broader work of psychological/brain engineering to get people used to comfort and stop creating devils in their head when they are comfortable, otherwise I don't have any issues with transhumanism itself.
senordevnyc 1 days ago [-]
If the people in charge of the AI companies understand the risk I don't know, but I think it's a good point that they mainly care about the money.
Can someone who thinks this about Altman explain to me why he has repeatedly declined to take any equity in OpenAI? I genuinely want to understand your theory of mind here. Do you think he's actually in it for the power, and for some reason thought taking equity would undermine his effort at accumulating power? It just makes no sense to me.
pona-a 1 days ago [-]
I enthusiastically agree. I've been on this ride since GPT 2 and 3, I did start out looking at the technology in a neutral or positive light. I did not think the stewards of this technology would be so irresponsible, its consequences would rival the Stalinist purges.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
nobodyandproud 1 days ago [-]
I’m sorry, but the AI workers are as much to blame.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
i2km 1 days ago [-]
An excellent comment. It's very true - there's a deep sickness here which causes people to flee from the inherent value (relatively modest) and the longer, more tedious work of building a sustainable company around it.
CodingJeebus 1 days ago [-]
> there's a deep sickness here which causes people to flee from the inherent value (relatively modest)
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
bluefirebrand 1 days ago [-]
> It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
mcphage 1 days ago [-]
> Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
asdfsa32 1 days ago [-]
"We". Who "is" we? This isn't about you. or me. there was never a we. Before you open your mouth or put your pen down, wouldn't you think of the shareholder value for once?
swiftcoder 1 days ago [-]
Important to get your "Now I have become death, the destroyer of the worlds" on the record before anything untoward happens
hyperpallium2 1 days ago [-]
We might be better off without fire. Fire brigades, fire hydrant infrastructure, bush fires, destruction, death.
snypher 1 days ago [-]
Cooking food, surviving winter ..
fc417fc802 1 days ago [-]
"It might turn out we were better off not opening Pandora's box" says man who helped open Pandora's box.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
pixl97 1 days ago [-]
>predicting the likely outcomes
And if all the likely outcomes are very bad, then what?
fc417fc802 1 days ago [-]
Makes no difference. Figure out how to make the best of it. That's what it means to be inevitable.
You might as well ask "but what if all the options are bad" while standing on a sinking ship contemplating your next course of action.
motbus3 1 days ago [-]
Such a smart guy so late for the party.
Thanks to him and others who understand a lot about tech, we gave monster a weapon to destroy us.
I still don't know how Karpathy still works with those folks. I really think the guy is great but he makes me wonder
moezd 1 days ago [-]
Eh. We could've been better off by not inventing cars and creating wonderful concepts like jaywalking, highways literally tearing neighbourhoods apart, stupefying rush hour commutes, lead in gasoline... but who wants to get rid of their cars now, without any presuppositions about their current city's public infrastructure? The genie is out of the bottle.
1 days ago [-]
xnx 1 days ago [-]
This is a bit like Rudolf Diesel saying maybe we're better off without airplanes. The field has long since left this guy behind.
coldtea 1 days ago [-]
If the resulting issues are correct it doesn't matter whether the field left the guy behind or not.
He would still be right and the field practioners would be wrong.
Towaway69 1 days ago [-]
So pumping CO2 into the atmosphere is a good thing and if someone does a backflip by recognising their mistakes that is a bad thing?
xnx 1 days ago [-]
All economic activity emits CO2 directly or indirectly. The question is are the benefits worth the cost, and could the process be more efficient. A molecule of data center CO2 doesn't affect the climate any more than one that comes from producing cement, or getting a Labubu delivered overnight. If we want less CO2 emitted, it is best to treat emissions equally and let people decide what they value when they spend their money.
reverius42 1 days ago [-]
We should regulate data centers to require fully renewable power (solar/wind/hydro + battery).
vixen99 1 days ago [-]
This is tangential encouraged by your point and other forays here. But one does wonder what arguments politicians (the power lies with them) understand when they make their pronouncements. The UK has for instance, prior to the new administration, totally and vociferously supported net zero. From the point of view (a fact for many but not for all informed authorities) that climate change is caused by CO2 and we are headed for a disaster - Net Zero is a logical step to trying to save humanity and the whole ecosystem. Now (November) I read that the UK powers that be, appear to have put it aside in actions at least. Countdown to Net Zero doesn't matter any more; open the oil fields, get rid of subsidies for photovoltaic and wind - and other measures. My point is not a political jibe but a stance I suspect to apply in other areas. Is climate change going to be a catastrophe or not? They should make up their minds.
bigstrat2003 1 days ago [-]
We are already at a point where we were better off without genAI. It does a lot of harm, and not much good at all.
spwa4 1 days ago [-]
I guess capsule network haven't improved ...
sota_pop 1 days ago [-]
I actually laughed out loud.
I do find capsules interesting though.
insane_dreamer 18 hours ago [-]
> Hinton, 78, said that within a decade and possibly sooner, humanity will know whether it can manage the technology’s risks and enable what he called “radical abundance” or whether “we fail to manage the risks and … have a huge gap between rich and poor, huge cultural problems, massive unemployment, just society being torn apart”.
lifting people's standards of living -- is _not_ a technological problem. So I don't understand why people think that AI might solve this, even more so because AI is not primarily being developed or used to solve human or social problems, because there is little financial incentive to doing so (not much ROI in developing something that's cheap or free).
The Earth isn't lacking in resources, they're just distributed highly unevenly -- and correcting that distribution is a social and political problem, not a technological one. In fact, AI is more likely to exacerbate inequality than reduce it.
Let's say that AI allows us to crack the code to nuclear fusion. That would be amazing from a technological point of view. But the idea that this will lead to free, or even cheap, energy for all is rather naive -- all we have to do is look at whether the invention of efficient solar power (a limitless resource that is now relatively inexpensive to develop) has lowered energy costs for the average American [0]
Why on earth did he develop the technology only to spend his last years telling us how fucked we are because of it.
Like it would be fucking awesome if he could try find some more positive takes because it's tiring here a guy who basically got us into this mess, whine about how screwed we are.
Like good on him trying to raise awareness but I just find the guy so frustrating.
librasteve 1 days ago [-]
modern day Mary Shelley
1 days ago [-]
readthenotes1 1 days ago [-]
Interpretation: Geoffrey Hinton doesn't do anything to get it done.
He doesn't search the internet to find an answer. "Search" results are nearly useless but the AI load does a good job.
He doesn't lean on AI to synthesize large information.
He doesn't use AI to build spreadsheets.
Etc.
Fortunately, he probably also doesn't use AI for therapy which two of the already crazy people in my family do.
ftchd 1 days ago [-]
What is the point of this quote? Or any of his quotes for that matter.
FLeXMurphy 1 days ago [-]
Clickbait, including the photos taken of him with various facial expressions. It all seems so trite and staged.
dboreham 1 days ago [-]
I think a big underlying issue was that nobody thought AI would work. I certainly didn't, without some magic sauce quantum processor, or 100 years of Moore law. So nobody did much thinking on the consequences. I didn't begin to pay attention until my non-technical friends began sending me 10,000-line Rust programs they had written.
deadbabe 1 days ago [-]
We should introduce regulation so only credentialed software engineers are allowed to use AI.
harvey9 1 days ago [-]
I would rather see the onus on the company owners to be responsible for what they sell. Serious penalties for seriously bad products.
jdw64 1 days ago [-]
In East Asian countries, certification systems are still maintained, but in reality they're practically meaningless. If a certification system exists, people will take an interest in CS and work harder at it, or so you'd think—but in reality, the exam questions end up stuck in the old style. In other words, there's a big gap between the exams and the actual field.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
carlosjobim 1 days ago [-]
Absolutely! And software engineer should be an hereditary title as well.
sneak 1 days ago [-]
It might turn out we were better off without nuclear weapons, but so far I’m happy we haven’t had another world war (in the traditional “total war” sense).
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
coldtea 1 days ago [-]
>so far I’m happy we haven’t had another world war
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
bob001 1 days ago [-]
> roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
coldtea 7 hours ago [-]
>Both the USSR and US fought multiple wars directly against the proxies of the other.
They did, but didn't directly go for war between them or raised it to a 3rd world war, which is the point.
>In most any other era one of those would have become a full blown war between the two.
Would it? French and British empires were quite content not to go to war between them after Napoleon, when they felt they had their hands full with their own colonies each. After 1815 they've didn't do it, and even were on the same side in wwi and wwii.
insane_dreamer 18 hours ago [-]
I would serious debate that it's the threat of nuclear MAD that has prevented WW3.
Rather, after WW2 the idea of millions of people dying in a war became anathema -- countries would simply revolt against the idea.
Also, an estimated 20-40 million people have died in wars since WW2 (depending on how you do the counting), so it's not like wars have stopped.
1 days ago [-]
drix10 1 days ago [-]
> mandatory testing of frontier AI models, independent evaluation and intergovernmental common standards
The FDA analogy is structurally wrong for AI. Drugs have a fixed chemical composition and a testable biological mechanism; the FDA approves a specific molecule for a specific indication. A frontier model is a weight file that can be fine-tuned, quantized, distilled, or used with different prompts and toolchains after release. Pre-release testing cannot anticipate every post-release adaptation. The thalidomide comparison also misses that thalidomide's harm was a single, repeatable teratogenic effect. AI harms are emergent from use patterns that don't exist at test time. This is the core tension that makes any pre-release approval regime unworkable without also controlling how models are used after deployment.
vedani 6 hours ago [-]
[flagged]
catoc 1 days ago [-]
[flagged]
pixl97 1 days ago [-]
[flagged]
catoc 1 days ago [-]
I fully agree with the latter points, but are you sure you are replying to the correct post?
bondarchuk 1 days ago [-]
So why did he start to develop it in the 1970s?
emil-lp 1 days ago [-]
Why did he start a research direction that 50 years later "might turn out" to not be a good idea?
That's a relatively silly question, don't you think?
bondarchuk 1 days ago [-]
No it's a completely serious question. The goal has always been to replicate human/biological thinking processes in software, and now that this has been more or less achieved he's suddenly not happy about it. There's been plenty of opportunity to actually think about "what if all of this actually works" but they chose not to.
bigstrat2003 1 days ago [-]
For one thing, we haven't remotely achieved that. GenAI is a statistical model, it cannot think or reason. So one reason he might lament the status quo is because it has people thinking we have achieved AI when we haven't. But also, it wasn't apparent until recently that even if the tech did work as promised, it will be an awful thing because humans will use it to bad ends. I don't think it's particularly blame-worthy that he didn't foresee that 50 years ago.
bondarchuk 1 days ago [-]
I don't think we need to have the "can AI really think* debate for the millionth time, anyhow it's safe to say that we have progressed towards the goal far enough that Hinton thinks it's a problem.
>So one reason he might lament the status quo is because it has people thinking we have achieved AI when we haven't.
I don't think the article allows us to infer this at all.
>But also, it wasn't apparent until recently that even if the tech did work as promised, it will be an awful thing because humans will use it to bad ends. I don't think it's particularly blame-worthy that he didn't foresee that 50 years ago.
No, really, that would be extremely naive, we can expect better from scientists.
injidup 1 days ago [-]
Sounds like sour grapes. He worked on it for a long time but he doesn't own a trillion dollar company. Missed the boat?
dude250711 2 days ago [-]
We should have not left the trees. You cannot disrupt banking and power if they do not exist. Checkmate!
olelele 1 days ago [-]
The ghost of Douglas Adams has entered the chat
jjgreen 2 days ago [-]
The scene is bleak, silent, desert-like with a few dead trees bleached white by unfiltered UV. Nothing moves. In the distance a cave, and from that cave
... no shit Sherlock
bluebarbet 2 days ago [-]
9 minutes. Was that enough time to read the article?
catchnear4321 1 days ago [-]
Did you look at it? It wasn’t that long.
ionwake 1 days ago [-]
I personally feel reading the article before commenting affects the purity of my vibe before posting
librasteve 1 days ago [-]
vibe comments!
emil-lp 1 days ago [-]
My reader mode says 6–7 minutes, so, yes.
TFA could have been two sentences:
He warns that AI might cause serious problems: power failures, job losses, and greater social inequality.
He believes AI might improve our lives, but governments must make sure it is safe.
neonnoodle 1 days ago [-]
oh word?
throwthrowuknow 1 days ago [-]
[flagged]
Rendered at 22:55:10 GMT+0000 (UTC) with Wasmer Edge.
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.
Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.
Are they?
Consider the old joke, "what do you call the medical student that graduated bottom of his class? 'Doctor'".
Same thing applies to sysadmins and IT. Sure there's plenty of skilled people working in facilities out of bumfuck, but also consider how few people want to work in the low-paid, low-respect IT jobs.
Those roles are often filled by the apathetic, mid-skilled, and close in proximity. They're connecting shit to the internet because their boss wants more data points.
What makes this particularly embarrassing is that power companies literally run thousands of miles of cables and could easily build a private network instead of using the Internet. The regulations should be amended to require it.
Again, STUXNET shows that bugs get into authorized devices really easily. One of the biggest issues is you have power guys that don't know dick about computer security using fleets of laptops that will need a huge amount of training to bring them up to the needed high levels of security.
More so there are thousands of these devices out in the field in power company trucks. All it takes is someone picking one of these up and infecting it for the problems to spread.
It is an insanely hard problem to solve. Breaches happen in military networks pretty often. As a measure of secrecy, they never tell you about it.
While you certainly make a good point about the difficulties that come with securing the laptops and other equipment, the military has the same problem. Most of the users of classified military systems are not themselves experts in computer security. The necessary training is not a graduate level course in security; it is training people to always match cable colors to port colors, to only use laptops provisioned by the organization, etc. The military disables, sometimes by physical means, USB ports on laptops that are used on classified networks to close one of the key security holes Stuxnet took advantage of.
Sure, it is challenging, but this is not impossible to solve. Utilities would have to be forced to do it and they would need time to overhaul their operations, but they certainly can do it, and there is an argument that the NSA's IA division should help (as the electrical grid is a critical for national security). It can be done, we just have to be willing to do it (and pay the cost of doing it).
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
If we're at the point where all that matters is scale and reinforcement training then the public facing aspects could entirely disappear and progress would still continue especially when given an infinite budget of taxpayer dollars.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
The OP didn't say we should make it happen so I'm not intending to put those words in their mouth. Its a small leap from "it might be a good thing if the power is out for days" to someone hearing or thinking that themselves and acting on it.
If it turns out we have a power outage, people die, and we learn a lesson and adjust course then so be it. That would be what happens in that future but it isn't necessary and it isn't a good thing even if today we think it leads to the best outcome - we can't know that without knowing all the outcomes and would be generalizing into only the ones we can imagine.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
We both don't know all the possible futures and we don't know if a power outage or similar would kill 1,000 or 1,000,000. We also don't know if that loss would lead to the future we want.
Now supermarket fridges are openly accessible over the Internet
I’m not sure it’s AI that’s the issue, it’s just a scapegoat for decades of bad security choices that nation state bad actors could have always exploited
Now that’s a perceived threat, ‘AI will do bad things’ is a great excuse for awful national infrastructure and security management
If someone manages with the help of AI to hack into a vital system, then the system was already in a critically bad state beforehand. If developers can't use AI to check the code for security flaws, then security gets worse and not better.
Exactly. This is the same type of sentiment as security through obscurity that Hinton is advocating for.
Insecure is insecure. These systems never were secure.
You’re placing blame on the wrong spot. It’s algorithms and social media incentive systems not AI. Our digital media incentives addictive minimum effort chum and slop. AI just makes it easier to make.
> Human slop is slow and easy to contain and build containments around it. Generative AI is so fast that it breaks every possible containment strategy
If you think society prior to ChatGPT was any less off-the-rails than it is now, then you're suffering from recency bias. I'm willing to give you that AI certainly accelerates the slop, but it doesn't fundamentally change anything about humans being unable to regulate their consumption of slop.
But also no, the crap AI is causing goes waay beyond algorithms and social media. It is ok to blame it.
I think all these laws/theories etc needs an adjustment to today's reality. Human slop is slow and easy to contain and build containments around it. Generative AI is so fast that it breaks every possible containment strategy
A snowflake is not a problem.
A lot of snowflakes can be an issue as the pile up.
A lot of snowflakes rushing down a mountain at 100 miles per hour is a big damned problem.
Having AI in my life is a hill I will die on. I will not want to go back to being dependent on having other people do for me what AI lets me do for myself.
Let's flip your last point: if the tech gets good enough to replace people at any task (including physical ones), do you think others will need you for much?
And then there's the systems that the corporations are making more powerful as fast as they can. Yes, they promise to do things like cure cancer, but they're also capable of scarier things like hacking banks. And they're progressing faster than we can understand them. Would it be so bad to delay the cancer cures by a couple years to ensure we still understand what the models are doing?
With AI-driven speech recognition, I have far more agency than I did without, period. I can write, I can code, I can communicate with various entities, people, government organizations, corporations, chatbots. I can fill in online forms instead of leaving a bunch of spastic scrawls that generate more questions. I can use it to drive OpenSCAD and 3d print something.
And here's why I start out by saying I don't care if others need me.
I think I'm congenitally incapable of not building/designing/exploring things. If we had an AI-dominated future and we were all on UBI, I'd still find myself saying “hmmm, I wonder” and going down a rat hole to see what I can find. I'm doing it for myself. My curiosity and satisfaction in bringing something to life, even if it's only a variation on what somebody else did.
Rephrasing something I said in a comment somewhere that I would spend time gardening, bird watching, and building a closer relationship with my partner and my pets. I would have no want of things to do or explore.
To your last point, we don't need to understand how something works for it to be useful. Doctors don't understand why treatments work and why they fail. If doctors waited, if treatments waited until they understood how and why it worked, I'd probably be dead. But because they don't, I now have a VP shunt in my head (aka, medical-grade juice-box straw), and my life has drastically improved. Toddlers are something parents don't understand—the how or why they do what they do—but they keep having them anyway.
WTF moments are an emergent property of toddlers, goats, border collies, working dogs, hunting dogs, and AI agents. And like with a toddler, we need to deal with it like we're the adults in the room.
Too many people in my life have died of cancer, and the agony and pain as their body hollows out to a waxen shell is something I'll probably never forget. If I had a choice between understanding how AI works and a cancer cure, I would choose the cancer cure, hands down.
One thing we do need to do is slow down the development or the build-out of AI data centers and put a resource constraint on the model companies, forcing them to get better with the same resources. Sort of an economic “you can't have dessert until you've eaten your vegetables” moment.
I really got to sit down and write that blog post on why agents are like toddlers and that toddlers should have figurative machine guns.
If one doesn't provide incremental value, how do they get food, shelter, and compute?
Perhaps we'll be lucky like horses, and be lucky enough to become pets once our population collapses.
Without accessibility tools, there's not much incremental value I can provide. When the RSI showed up, I was advised by many people to spend the rest of my life on SSDI. I refuse to do that, but it was difficult to show I could provide incremental value. Yes, my mind still worked; the designer, the architect were still there. I had speech recognition so I could write, but a common reaction was, “How can you do anything technical? Your hands don't work.”
Now to the general case that you seem to be worried about: that AI will remove everybody's ability to provide incremental value. No worries there. The entire system will collapse. Labor has no value. Without labor, capital has no value. And we're all jolly well fucked. Now, Your value is how good you can become at asymmetrical class warfare.
At least that's where the current nihilist fantasies about AI leave you.
The AI nihilist view works when you have the wrong enemy. AI is merely the puppet, the distraction, just like the H-1B visa, offshoring, undocumented workers, etc. All these distractions are so that we won't notice the billionaires are enshittifying society and won't seek to dethrone them.
IMO, I argue the fix involves returning the tools of production to the hands of labor, and taking power away from capital. But as with many problems in life, the solution is left as an exercise for the reader.
I sympathize with "dethroning the billionaires" but to me it looks a lot like powerful, general AI systems are the most capital enhancing technology we've ever seen. Wouldn't the class struggle benefit from slowing down frontier AI development while diffusing local models to ordinary people? I feel both can happen at the same time.
As to the point of AI, it's fuzzy. We have a tool we don't understand how to use; it's incredibly valuable in some spaces, destructive in others, but we really don't know how or where to use it. We are trying to fit it onto the old model of work and labor value creation but don't yet understand what the new model should be. It's analogous to generals fighting the last war.
General AI systems: capital enhancing? I can understand how you and many others see that, but right now it is capital-consuming. Capitol is desperately looking for that model they can wield to get more capital out of the system than they put in. With the opinion reports describing frontier AI companies' bond offerings as junk, it looks like asset gamblers with capital are trying to unwind their exposure because they don't see how they're going to get returns.
I suspect the meme of using AI to reduce or eliminate labor is a continuation of capital's hostility to labor having any power at all. It has traction because capital sees reducing labor costs as a way of extracting value from their investments, and, as I said, they can't see any other way to do it.
Slowing down development won't make a difference to the class struggle. There are many parties in this race, and handicapping one does not slow down the others. However, slowing down deployment, taking time to figure out the right way to use it would greatly benefit the class struggle. Denying AI companies' data centers is probably the most effective way to slow them down. If they don't have the resources to spend tokens on answering your email, they will find more valuable ways to spend tokens that should benefit all of us.
Denying them data centers will free up resources, which will enable us to create distributed AI use. Altman's view that all the RAM belongs to him because it's more efficient is a valid model if you are trying to fit AI into the way we used to do things. Progress and new economic development seem to emerge from the edges and not the central power structures of an economy. So if we want to find that new model of how to work with AI, it is imperative that we can build out local models in the hands of many people.
I'm reading this in the context of the article currently at the top:
AI is forcing us to pay off 40 years of technical debt
https://news.ycombinator.com/item?id=50364336
Is it bad we're finding all the bugs now? Or is it good that we're finally being forced to make our critical systems actually robust?
I guess we could look at it as going through a world-wide Code Red...
Programming has never been this awesome.
Also the future where there is no sacrifice to achieve your lifegoals seems just unnatural to me.
I am building personal software for family (home automation where it is not possible, time capsule (future letters for kids), study materials for kids and much more.
Alternative is, it would take me hours of manual work and I'm stuck at computer.
Yes, I'm in AI heaven right now that allows me to spend way more time with family than I would have been able to and still be productive.
It's no different from someone seeing their car as just a tool to get from A to B. That might be an understandable mindset for people on HN, whereas people at a car meet might find something off about that mindset.
Not to mention, sacrifice can come in many forms. The simplest example is, the time required to learn something or develop a skill. As long as people want to pursue things just out of interest, that's not going away.
(Unless we find a way to download the information directly into our brains, I suppose.)
Thank you, brave HN commenter. You've truly made the site a better place with your astute rebuttal.
Mortality.
My prediction is that some people will be able to use AI to create essentially infinite money inside their bank accounts (through hacking); the prices of assets will grow to ridiculous sums as the people with infinite money buy up assets, but yields on all assets would soon approach 0%; still, one asset is going to grow at a faster rate than all other assets (E.g. an AI company backed by blockchain/tokenized shares that are impossible to hack) and everyone is going to start piling their infinite money into that asset; every other asset will crash because of the rapid, exponential price acceleration of the top asset making every other asset unattractive by comparison. Then fiat money would cease to exist and shares (or crypto tokens) of the top company will be used as money instead. Then prices of assets would stabilize again; denominated in the new currency... Then after that, we will finally be able to start innovating with AI.
Not to say that they're unhackable or unstealable, but merely that they dont have a central weak point that if broken, destroys the whole system
Traditional finance has the benefit that they can more easily reverse malicious transactions, but if hacks become too common, it will be impossible to correct fast enough and in a reliable way. With traditional banking, a hacker only needs to hack a few nodes to compromise the integrity of the entire system.
So yeah I also think crypto will win in the long run. I would bet on a tokenized stock with AI attached.
I think the top asset will combine the AI which is best at hacking with the blockchain which is hardest to hack. Ability to corrupt others whilst maintaining perfect integrity yourself.
AI for offence, blockchain for defence.
I don't buy this. There isn't some kind of voodoo "rogue AI(s)" out there which could randomly decide to take over their vendor stack or brute force their way coming into possession of the most critical infrastructures on earth. These kind of predictions could only be realized if a tiny minority who owns a huge amount of compute and a particular agenda decides to do this kind of thing, which also will require insane amount of data center capability to cause, especially some kind of financial "collapse".
Or some other government setting an AI to intentionally hack and crash the US?
>possession of the most critical infrastructures on earth.
I think you may underestimate how poorly we protect these things. They have all the same security flaws as our internet connected systems. They aren't written in some super special language, it's all C, C++, JS and other non-memory protected trash.
I decided long ago that - when there are plausible arguments either way - it's better to be an optimist. Optimism does not mean ignorance or denial; rather, optimists should acknowledge risks and problems while having confidence in our ability to generate solutions.
One of the biggest challenges facing us is a critical mass of pessimists. Not only regarding AI, but also with respect to the economy, democracy, liberalism, etc. And while these negative views are plausibly correct, they also act as a self-fulfilling prophecy in that they rob people of the agency that is needed to solve problems.
Seriously? It seems to be causing more problems, giving us more boring work, and less free time.... sounds like absolute bullshit.
We need to build it as fast as possible because it could cure all disease. Ignore whether that's possible, how it might happen, whether all we're missing to do it is more intelligence, etc.
The only way I see us curing all disease is to throw out all the names we've given to collections of symptoms. What disease? I don't see no disease, you just have a sore throat, runny nose, and mild fever.
Hell no. We need to risk pumping out even more CO2, causing more environmental disaster, ruining entire human communities, just to cure diseases? You might as well say it's better to be in prison because you're guaranteed three meals a day. The logic of some pro-AI people is baffling.
The other day I hired an accountant and even he sent me an AI answer that sounded right but was actually bullshit.
In short, it takes a LOT more time just to find some sanity now.
AI is shit.
How so?
Only 14 years old and accelerationists think we can just push the capitalism pedal further and somehow get social changes going their way
Looking back now, was it really happier before AI existed? I'm not sure. In the past, when I wanted to find out whether I was right or wrong about some piece of knowledge or information, I'd contact several people and, far from getting an answer, I'd often be told not to bother them. But AI, in that respect, is the best teacher I can think of.
I think the times are continuously getting better. It's just that there's a problem of power concentration among the people who handle it. Still, if AI is a tool of that nature, then I suppose that can happen.
But considering what access to knowledge was like before AI came along, I think having AI is better.
AI has the benefit of having trained on all that material, it's a snapshot in a sense. But as people increasingly stop creating content like that and we turn to AI instead of asking for help on forums we stop growing the knowledgebase that AI draws from.
I don't think forums are bad, but I've questioned whether experiences in such forums were always pleasant. There was a lot of wrong information, and there were also many people who picked fights for no reason simply because they didn't like something. But AI really does just tell you knowledge easily.
Of course, forum activity gave me the opportunity to come across knowledge whose keywords I didn't even know, and that got me studying, but I question whether such forum activity was open to everyone.
I also somewhat agree that the problem of the knowledge base shrinking is real. But is that problem really unsolvable? As far as I know, the results on the degradation problem of synthetic data say that if there is synthetic data, there isn't that much degradation. People, too, tend to idealize the past as they get older. Like treating the inconveniences of that time as nostalgia.
I don't know. Rather, once the current SLOP era passes, certain knowledgeable people may become rare and be treated well then. The future can't be known.
Anyway, this is purely a matter of values "Some people may regret the destruction of knowledge communities, but I benefited from that destruction, so I can say it's good. People always judge by their own experience, don't they? I respect your past experience. In the cases you went through, I think AI produced more bad effects than good.
I've also seen many things disappear because of AI, but it let me try things I wanted to try, and I think that part is good. So, honestly, I don't know. I have both affection and hatred for AI. But the affection seems a little bigger.
And certainly dealing with people online can result in unpleasant interactions. I found that specialized forums were usually well managed and wouldn't tolerate abusive members, though new members were expected to learn the forum's rules and put some effort into finding existing info before they start asking questions. The downside of AI is that it can only tell you things that are already visible online, and it will take the stance of the majority. On a forum you would get people who can chime in based on their experiences where there's absolutely no prior info online, or people who could contest the majority and explain why.
Synthetic data certainly could solve this. Especially if we get AI controlled robots or AI managed physically accurate simulations of the world, such that AI can actually engage with the world and test new things. It will take a long time before we get to that point where AI is able to reach the level of being able to tell you something like "oh yeah if you take apart that DSLR lens, the screws holding the floating group in place are glued in with loctite. And for good reason, if you don't have an optical bench you're never going to be able to get those elements recalibrated if you adjust them. But it looks like your copy has a shim behind that group - try removing that and see if that lets you focus past infinity." Like AI can understand a lot of that in the abstract, and might be able to guess at some of it, but until it's actually pulled apart this lens and messed around with it it's going to be vulnerable to just making things up or simply not knowing key details. But that's why it does so well with code, since all code is accessible to it.
I do agree that AI is useful though, and I have no qualms about using it to learn things faster, build tools that help me, or do my job. And though AI has contributed to the loss of the internet I value, it's not the only contributor, and likely not even the main contributor. Search engines were already consolidating power into a smaller subset of large commercial sites rather than surfacing forums and blogs, sites naturally went offline over time and archiving efforts just weren't sufficient to make up for this, and users just prefer lower effort forms of engagement like scrolling Reddit with everything in one place and lots of entertainment rather than being a member of multiple more focused communities. So while I may malign AI for the impact it has had, you're probably right that on this topic the net impact is positive.
I also agree that specialized forums have well-managed aspects. But conversely, getting into a specialized forum usually required a ticket of admission—something like having to attend a certain conference socially.
So the problem is this point. Very high-level forums required an admission ticket, or basically a certain level of qualification, and were hard to get into; you had to get in through an existing member, and if you were physically far from that member, it was hard.
Usually, the forums someone like me could get into were often ones at a somewhat lower level. These days things move to Discord, but it's hard to go into Discord's closed ecosystem and study there.
Actually, as you said, forum activity and human interaction are important. Because in the end, humans have to learn about the world through humans.
The world isn't simply a place to obtain information—things like the patience to endure unpleasant situations and people, the understanding to listen to others' opinions without distorting them, and the courage to admit to others what you don't know are not things that can be cultivated through AI, I think.
The various values that have limits when learned through AI can be learned by mixing with people. In that sense, people have to learn about the world among people.
Because AI basically gives you wrong information to please you and reinforces information only within the scope of what you already know. I think friction with people is necessary to find keywords you genuinely don't know.
My words got sidetracked in the middle. I agree with your comment that AI isn't the only thing that changed the internet this way, and I also agree with your point that forum activity and the like are necessary. It's just that where we put the emphasis and weight is what makes our thinking differ. Thank you for the long comment, and have a good day
Most people are not interesting in learning for learning's sake, they are motivated to learn something they can apply (see: kids complaining about how learning Math in school doesn't transfer to useful skills). Why learn about architecture and technical debt, if you can tell the machine to make it faster?
I basically don't think you're wrong. However, books and reading have their information fixed at the point of publication, whereas depending on where you live or your community, the information you get buried in differs. Because I was in an Asian country, it was actually very hard for me to synchronize my programming skills with the latest technology in the US. The frameworks commonly used in my country are stuck at roughly 15 years ago. The time lag between US technology and my mainstream technology is about 5 years. In my area, it's about 15 years.
Anyway, thinking about your opinion, I think there are both valid points and dark points. I think learning for its own sake is important. I agree that people should have an attitude of learning for its own sake, but I don't think it's mandatory. I think it should vary depending on the environment. Still... I do think having that attitude is a good thing. I like your idealism, senior.
However, that holds for people who are middle class or above and can survive. Because rich kids can picture the future and are given the right to love learning. But if you're poor, that's very hard. And when you grow up in such a poor family, there's an emphasis on it having to lead to practical skills.
And even if someone loves learning for its own sake, I think quite a lot of people love the honor that comes from learning. But that too can be called loving learning itself. It's purely a matter of individual values.
I don't think you're wrong, senior, but I also wonder whether the old ways are right in today's era of excessive information. It's just that because our lives and the environments surrounding us are different, we came to different conclusions. I upvoted because I agree with your thoughts. But there are points where I differ. Have a good day.
Uh... we're already at the latter prediction.
The carelessness is also clear in how AI is rolled out. It's being presented as a disruptive technology, as is everything else presented by modern tech companies. The goal is a very large payout in a very short period of time, it's not to make anything better, or worse, either is just a side effect.
Had LLMs first been presented as a "better IntelliSense", where rather than auto-completing function names, your editor can now suggest implementations, very few people would have been upset. Same for writing, it's just a better version of Grammarly, it's fine, or as a tool in drawing software. The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
I'm not rooting for AI companies or hustlers that are out to ride the wave and get rich.
However your claim is just hyperbole. I've used it in cross-cutting projects to extend protocol on desktop, iot device, which involves holding premise long enough to co-develop and test. It is also fantastic in debugging multi-threaded systems.
Some times it comes from people who try out ChatGPT free tier or the Google AI summary and think it represents the state of the art.
More recently I suspect it’s from people who build their opinion based on being contrarian and cynical: They dislike the AI people and distrust everything they say. If the AI people say it’s changing the world, then the cynical and contrarian belief is to assume that the technology is actually very bad and they’re just lying to make a lot of money and retire.
These takes were more understandable in the earlier phases of AI when it really was very limited, but it’s getting harder to understand how someone can be thinking of AI as just a slightly better autocomplete or slightly better Grammarly at the end of 2026.
That wasn't the point. My point was that if you had rolled LLMs out like that, as embedded technology in existing products. Slowly, over time, you'd have a lot less cynical people. I think the hostility towards AI product in large parts comes from the disruption and glancing over the negative parts. What we're doing now is the equivalent of putting millions of cars on roads which five years ago had never seem a car, only horses. We're doing so without any understanding of how that might affect society. Social media was a similar story. It was slower, yet we still didn't catch the majority of the negative consequences.
If it turns out that the people behind social media, and smart phones, don't care about e.g. peoples mental well being, then why would I trust that the people behind AI wouldn't behave the same way?
1. enhanced autocomplete -- either line (e.g. JetBrains small language models) or longer predictions;
2. chat -- asking questions related to the task you are doing;
3. tool assisted/agentic -- letting models interact with tools to perform tasks.
I personally find 1 & 2 helpful for my development tasks. For 3, I'm cautious of using it for several reasons:
1. lack of control of what gets sent to the model (e.g. if you have confidential user data, proprietary code and/or data, access tokens/private keys);
2. letting the models run arbitrary commands (modifying the system, deleting files, etc.);
3. not having control over the design process, formatting of the files, etc.
Then the Chinese came along and showed that you can take those model weights, repackage them as open weights, and undercut the business model. Suddenly Dario et al. start talking about how dangerous AI is and how it needs to be regulated right now, before someone's abliterated open-weight Chinese-provided business-undercutting model destroys the world. They want their walled gardens regulated into being the only game available.
Both stages of the game—raising capital and regulatory capture—require hyping up the disruptive aspect of AI.
Presenting AI as something it isn't doesn't change the reality which is like the Wait But Why thing https://waitbutwhy.com/2015/01/artificial-intelligence-revol...
That was written in 2015 before LLMs were much of a thing and has been fairly spot on so far. This is not just about LLMs any more than the industrial revolution was just about steam engines.
The AI companies are as much being pushed along by investors as they are in the driving seat, but the push is in the direction of getting real products in front of people and experimenting with this miraculous tech everywhere it could possible work.
Ignoring the implementation details, if we treat the whole global technology world as one single AI system made up of AI companies, independent developers/researchers, and end users, what would be the training goal and evaluation?
How many, approximately?
It seems obvious that this same dynamic is in play with the frontier AI companies now. They know if they win the race to the they'll be forgiven their sins, or anyway they'll be beyond the reach of effective punishment.
https://en.wikipedia.org/wiki/Worse_is_better
Aside from the AI dungeon app, this was my first real exposure to LLMs wrt github's copilot extension. It sucked, and did a lot of stuff wrong, but they got exactly 5 bucks from me to try it.
It's sorta crazy to think how many years ago that was now.
They certainly don't care when a model hallucinates or an agent does something it shouldn't be doing. It's your fault and they charge you for the tokens.
That’s why the AI heads all have extremely dramatic narratives. They are the end product of a selection tournament for raising massive amounts of money pre-product.
You make it sound like this was a feasible option. This technology would exist regardless of how any one company decides to frame it. If OpenAI decided to go with a more conservative approach rather than just throwing ChatGPT out for the masses, then all it would take is for one person to realize they didn't take full advantage of this technology to just make their own ChatGPT.
>The issue would have been, for the AI bosses, that this takes time and doesn't create trillion dollar companies. It only creates companies that they'd actually have to run, long term, and that's boring.
You can criticize the current tech leaders all you want, but what you're describing is a fundamental dynamic of capitalism. You must choose the most profitable option, otherwise you have chosen the wrong option. If the current tech billionaires all vanished just as ChatGPT was released, they'd be quickly and easily replaced with roughly equivalent individuals who would effectively operate the same exact way. That's true because they're operating based on the mostly deterministic policy of profit maximization. They don't really get a say in it; they must follow the path towards maximizing expected future returns, otherwise they'll get canned by the investors who are seeking maximized expected future returns.
None of this relieves them of their personal ethical responsibilities either. I'm just pointing out that your comment might as well be describing an alternate reality where everyone lives peacefully with each other and nobody goes without what society can afford to offer them. It's equally as unrealistic.
I feel like you’re not acknowledging the inevitability of the technology being developed further. You’re also ignoring competition and the basic fact that people do research and push the boundaries of their field naturally.
If the major frontier AI labs had decided, as you said, to make a fancy autocomplete and a better Grammarly and then simply stop and coast, do you really believe that every AI researcher and entrepreneur in the entire world would look at the situation and decide that the status quo is where it should end?
Humans don’t work that way. People would get ideas and want to push the limit. If the frontier labs stopped developing their technology, other companies would be created to carry on. Academic researchers would push the limits further. It would be discovered that the AI could go further and we’d get back to this same eventuality with a different set of people leading it.
I also disagree with your characterization of this as a get rich quick scheme for AI leaders who want to make a lot of money and then be done. I think this characterization feels truthy if your only conception of startups is as a gamble to retire early, but the people who build and run these companies actually do like building and running companies. If the AI leaders and executive management you’re talking about were separated from their jobs today, even with billions of equity, most of them would be out starting new companies next month because that’s what people like that do. I’m not saying this to “glaze” them or whatever I’ll surely be accused of in the comments, I’m stating some facts about this space. You don’t need to look far to find examples of this already happening.
Despite this, when discussing the egregious behavior of the frontier labs, there is vanishingly little acceptance of the idea that 1) yes, it probably was inevitable in some ways AND 2) those involved, particularly at the top, are still responsible for the consequences of their (bad) decisions.
The models we currently have aren't going away, and especially the ones small enough to run at home will continue to be used, but if these data center investments end up bankrupting Big Tech, I don't think we're ever going to see anything beyond that. Not for a long time, at least.
My guess would be that once the money runs out, things slow down, but incremental progress continues to be made. A focus on smaller models will improve their capabilities. We’ll get better at driving them. It’ll be used in places where it truly makes sense, instead of anywhere & everywhere just because it’s easy.
I don’t actually know what the financials are, but even without the effective subsidy from VC money, I have to believe there are things AI’s doing today that’s way cheaper than what it would cost to have a human doing equivalent work. And that dynamic will drive further innovation in the space for a long time.
What does this actually mean? In what sense are people using this term. In the most strictly literal sense then sure, nothing is inevitable, because all paths are open.
But in the practical sense, I am quite happy to use the term inevitable for many technologies. I think if ASI is possible (and I think it is), then it is also inevitable. I would happily work towards developing ASI technology, because I believe it will be ultimately good for the world. But it sounds like a lot of people want to lock me up or even kill me if I did that.
1. Is it both inevitable and still they are responsible? 2. Or it was inevitable and they were just an instrument to make it happen. 3. Or it wasn't inevitable and if it were not for some specific people then it would not have happened?
I am leaning more towards 2. But if that is the case why are only specific people reaping the benefits of this?
If it is 3 or 1, then why aren't we holding them accountable?
So yes, they are 100% responsible for damage they caused and did not had to happen. And no, "I had no choice, because it was the only way to become billionaire" is not an excuse.
They were NOT "instrument". No one manipulated them. No one forced them. Nothing forced them. They themselves wanted to and did not cared
Other labs seem to be working the exact same angles as them, but just have the good sense to not be so open about it. It's not the "lab" people have a problem with, it's their leadership. These people cannot be trusted to make the right decisions for society with this technology.
People are starting to panic, and few things bring out full blown idiocy and disregard for ground truth faster than widespread panic.
So yeah, I'm negative against their development of AI along with the general public, and for good reason it would seem.
Also, I don't want super intelligence. I want humans to remain in control. Why are they trying to build something to replace humanity? Why don't we got a vote in this? All 8 billion of us not developing frontier models with the expressed goal of achieving AGI followed closely by ASI.
It's a pity because as computer scientists this should have been such a fun technological moment for us. What a cool thing to experiment and explore new open doors with. But instead it was immediately the driver of mass job loss, societal upheaval, and a double digit percent chance of ending humanity, espoused by the AI lords themselves.
I do. Quite a lot of people do. And that is because we believe it will lead to a better life for most humans.
The list of possible reasons is pretty short and mostly not really good for most of us, as far as I see it
You must be joking. Altman is a sociopath, Amodei is either too stupid or too evil to stop working on tech he claims will have a 10% chance of destroying humanity (and in either case he should be imprisoned to protect the rest of us from him), and Musk is a man-child who will lash out at anyone and everything who dares to speak ill of him. There's absolutely nothing "lucky" about having these jokers in charge of stuff.
It's similarly easy to come up with "what Altman should do better". The dude is a genuine sociopath in charge of a company trying to make everyone's lives worse off in order to enrich him and his investors. It doesn't get much worse behaved than that.
This looks a lot like the “if I didn’t do $badthing someone else would have” line of reasoning.
I'm not arguing about the consequences or responsibility. I'm asking which humans would be capable of stopping the development of AI at this point in history.
It’s difficult to understand where this all is coming from because it’s so strangely disconnected from history. It’s not even consistent with current world events. The US is in a war with Iran over nuclear proliferation while also worrying about North Korea’s nuclear program. China and the US are in a trade war and Taiwan is under threat due to semiconductor capabilities (among other things, obviously). Open weight models allow anyone in any country to do AI work on their laptop. The top story on HN yesterday was a small team of academics making some cool discoveries on improving LLMs without frontier lab level compute.
How anyone can see all of this, in addition to decades of history of attempted nuclear diplomacy and wars, and posit that the whole problem is easily solved by simple agreements is confusing to me. The confidence with which these alternatives are being proposed and the unwillingness to acknowledge second order effects (like AI researchers moving to a different country and continuing to work) is baffling.
But yeah, there are a lot of comments from genie-rebottlers that are not very grounded in reality. They're functionally equivalent to saying "hey, everyone in the world, let's not actually use electricity after all" in 1905.
If we're talking game theory, a better model is the Stag Hunt [1]. At some point the equilibrium strategy is to cooperate.
Another difference is that real life is an iterated game, and the iterated prisoner's dilemma rewards cooperation.
[1]: https://en.wikipedia.org/wiki/Stag_hunt
If there is a room full of people, and one person gets a gun, who would you pick to have it?
Yourself, obviously.
Maybe america will end up the only world power holding the secrets of LLMs? ... Oh no...
If nuclear weapons were as easy to build as AI models are to run, we would have all been dead decades ago.
There might have been. But the notes from the Manhattan project were leaked almost immediately to the russians by informants.
Still seems a moral cop-out, and while true, the timeline would certainly have changed, if not the inevitability, if key figures such as himself had opted out.
> Nazi Germany undertook several research programs relating to nuclear technology, including nuclear weapons and nuclear reactors, before and during World War II. [...] The first effort started in April 1939, just months after the discovery of nuclear fission in Berlin in December 1938, but ended shortly ahead of the September 1939 German invasion of Poland, for which many German physicists were drafted into the Wehrmacht. A second effort under the administrative purview of the Wehrmacht's Heereswaffenamt began on September 1, 1939, the day of the invasion of Poland. The program eventually expanded into three main efforts: Uranmaschine (nuclear reactor) development, uranium and heavy water production, and uranium isotope separation. Eventually, the German military determined that nuclear fission would not contribute significantly to the war, and in January 1942 the Heereswaffenamt turned the program over to the Reich Research Council (Reichsforschungsrat) while continuing to fund the activity.
> The program was split up among nine major institutes where the directors dominated research and set their own objectives. Subsequently, the number of scientists working on applied nuclear fission began to diminish as many researchers applied their talents to more pressing wartime demands.
> [...] Heisenberg said that in 1939, the physicists at this second meeting concluded that "in principle atomic bombs could be made, but they also emphasized that it would take a number of years",[24] certainly not less than five. He also said that he "didn't report it to the Führer until two weeks later and very casually because I did not want the Führer to get so interested that he would order great efforts immediately to make the atomic bomb. Speer felt it was better that the whole thing should be dropped and the Führer also reacted that way."
> [...] Speer states that the project to develop the atom bomb was scuttled in the autumn of 1942. Though the scientific solution was there, it would have taken all of Germany's production resources to produce a bomb, and then no sooner than 1947.[39] Development did continue with a "uranium motor" for the navy and development of a German cyclotron. However, by the summer of 1943, Speer released the remaining 1200 metric tons of uranium stock for the production of solid-core ammunition.
There are ways of calling this "weren't working on a nuclear bomb", but it seems to me like "they wanted to, were moving in that direction, and would have done it if they had the resources and weren't distracted by shorter-term considerations".
That sounds very much like "they weren't really working on it". They explored the options and decided not to.
Whatever you want to call it, that's what I was referring to.
How is any of that good?
---
Simply put humans are at risk of any power scaling. We exist safely in a very tiny temperature and energy environment and the rest of the processes in the universe operate widely out of that strict safe area.
It's also a catch-22. There are large rocks in space that will eventually end us unless we do something about that. But having the power to do something about that gives us the same power to also end everyone on the earth.
Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]
And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.
[1] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...
[2] https://en.wikipedia.org/wiki/Nuclear_close_calls
> But why are we rushing so much?
Talking about these things with elaborate analogies only obscuring the point you already stumbled into: Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.
Talking about humanity like one entity that operates together misses the point: There are billions of people in the world and some of them are going to discover new things.
There is no general counsel where humanity gets together and decides what humanity works on. There is no singular bag with a singular “humanity” pulling marbles out one at a time.
The analogy is too broken to be useful for the discussion.
There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.
You’re speaking of “we” as if the whole world was operating in unison, while there’s a war going on in Iran over nuclear disagreements.
The nuclear situation only reached stability after multiple countries fully developed it. I don’t see the parallels to halting AI.
The nuclear stabilization that occurred was the result of a collective understanding of the existential risk it posed leading to anti-proliferation agreements and superpowers going to great lengths to prevent nations deemed as unstable from obtaining such technology.
Some portion of the global community may collectively discover/decide that the risk AI presents is worthy of anti-proliferation treaties and wars may be fought over it.
The main gap in the comparison is that we have a much clearer understanding of the risks of unchecked nuclear weapons use than we do about what current and future AI tech might or might not unleash.
The underlying point is that we have historically placed limits on our curiosity when we deem certain paths of exploration existentially dangerous. It has yet to be seen if AI will reach those levels of existential risk, but if they do, there is plenty of precedent for stepping on the brakes.
Nuclear Non-Proliferation Treaty (1968)
Biological Weapons Convention (1972)
Asilomar Conference on Recombinant DNA (1975)
We've done this sort of thing plenty of times around specific, dangerous technologies.
Also the bio/DNA ones are quickly falling apart as the technology to do these kinds of things on a small scale are readily available. To the point where we're finding illegal wet labs in the US.
Nuclear was more of an exception because of many of difficulties in purifying stuff for weapons and its detectability.
AI is a much, much, much harder problem is it uses the already existing infrastructure as a base. Unless we regulate, monitor, and ban all private usage of technology AI can be developed at a small scale.
This is true, but not all the time, and transparently not the case for the current leaders in AI development. Their motivator is money.
With AI, the point is who uses it for what.
Think about it like a magic wand that makes your wishes come true. Would you really want everybody to have one? Who should and who shouldn't? Who gets to decide?
AI indeed goes far beyond "autocomplete" and other nonsense. But the question is, whether it's a good idea to just let the "market" run wild with it.
Short-term capital interests don't align with the public good. AI greatly scales capability, but without anything to restrain it, that implies those non-aligned interests overwhelming society's defenses, outright destroying it in the process.
The idea of rich people being some kind of benevolent caretakers, who independently and dependably foresee and manage the consequences of their actions in everybody's best interest, is laughably wrong already.
With AI, that reality gap will swallow whatever is left.
Without capitalism, you wouldn't be able to restrain AI adoption any more than the armored knights of the middle ages were able to restrain the adoption of the firearm.
I operate within the logic of how the world actually works.
Which I find infinitely more useful for practical topics like the reality of AI development in a global economy.
Fantasizing about alternative realities where all the leaders of the world unite and halt technology development without any of them secretly defecting and developing it further isn’t very useful to discuss. These things aren’t going to happen.
Additionally, you don't need all the leaders of the world to agree to anything, you only need US and China.
Additionally x2, if you agree with the common stance that China makes the vast majority of their progress by distilling US models, you don't even need them to agree to anything.
You're really suggesting the following bans. A ban on computer science and algorithms. A ban on any device that can play games that aren't encrypted or DRM locked from an approval authority. A ban on any kind of unregulated computing without government approval and licensing.
>China makes the vast majority of their progress by distilling US models
Eh, this is a budgetary decision, not a technological one. Why spend billions when the US can spend billions and you can spend millions. China more than has the power to do what we're doing if they chose to do so, their biggest limitation is speed of making new compute, not money or brains.
You know what, fuck it, that's exactly what I'm suggesting. Let's ban personal computers. The only consumer hardware allowed should be a HTTPS client directly burned on the chip that can only connect to government servers. Absolute all timer of a straw man. This is probably the last comment I'm writing on this site, this place has become unsufferable.
Believe it or not, there already are things you're not allowed to do with computers that you otherwise own, and we have ways of enforcing them other than trusted computing -- the threat of putting you in a cell for a few decades usually does the job. That's without getting into the fact that you're not going to train 1T-sized models on your laptop, and billion dollar GPU farms aren't precisely hard to keep track of.
it has kinda worked for nuclear weapons
AI has no such hurdles.
That’s the inverse of what I said.
Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".
When you imagine people as savages to be outdone in their savagery, you become one yourself. Spend more effort on finding ways not to.
Ignorance of what? All of the previous successful attempts where the entire world reorganized itself into a single world government?
> Transitioning from the current "order of things" to one compatible with the existence of things like AI is a necessity, not a "fantasy".
I think you’re putting the cart before the horse. You’re assuming it’s necessary and then making another leap to assume that being necessary means it must also be possible.
I think the vast majority of the rich are greedy bastards, but IMO being rich does not imply greedy bastard 100% of the time.
The argument from nature is a fallacy. Just because something is doesn't mean it ought to be.
This doesn't seem right: it doesn't match the history of reality, in which capitalism has repeatedly stymied rather than fostered competition.
If it weren’t OAI and Anthropic, it would be others.
If it weren’t US companies, it would be others.
It is far too useful now to stop large scale deployment, it isn’t a matter of specific people or countries not being cautious or prudent enough. The entirety of the Earth is at issue now and there will always be people who cause the tragedy of the commons (if indeed AI turns out to be harmful in the long run, which is not remotely certain).
Look at what happened with fossil fuels. Even if the entire global west totally ceased use, there are many countries that simply will not stop because of local utility. AI is the exact same way.
There is something deeply rotten about the US mindset though, them being American is definitely a problem.
> Look at what happened with fossil fuels.
The west had a 100 years head start, on top of that these countries are producing everything you consume, don't start pointing fingers. Chinese current co2 per capita is the same as the US in 1900
What is that “something”?
The world’s biggest philanthropists are American.
The US has for decades invested into charities and human development around the worlds .
> deregulation at all cost
The US has regulations. The Constitution of the United States is the oldest and longest-standing written and codified national constitution in force in the world.
> Christian brained politicians
We have separation of church and state.
The Sackler family of opioid infamy were huge philanthropists. Not saying that the AI companies are like Purdue Pharma, but clearly philanthropy does not compensate for societal damage.
Americans and the US as a whole are the most charitable in the world.
The data does not lie.
Americans can be charitable individually, but that does not contradict the reality that US corporate and institutional structures operate on a strict "profit above all else" mandate.
https://en.wiktionary.org/wiki/Goomba_fallacy
Yeah because it includes lobbies and turbo billionaire pastors from texas using "charity" money to buy rolls royces as well as other highly retarded """charities""" that aren't even legal in the rest of the developed world
You have senators and wannabe senators quoting the bible on TV as their guiding star as we speak right now, give me a break
On top of that you can separate the church and the state all you want, when people like Thiel show up it doesn't matter, these people are deeply troubled, if they were muslim and saying 1/10th of what he says you'd send them to guantanamo
Releasing artificial intelligence for the use of all of humanity is better than it being controlled by one or more shadow groups. That's what the Americans did, and the Chinese followed in.
Had AI been invented in the 20th century, it would have been kept secret and used only by state actors to facilitate genocides at mass scales. But we live in different times.
How could modern AI have helped those groups you mention in their endeavour? Really fucking easy and effectively, and you don't need a lot of imagination to think of how. From surveillance and interrogation to pinpoint determination of ethnicity and much more. Not to mention translation between human languages.
They literally targeted people for their past opinions, how they looked, etc. which you can now determine and process at break neck speed, you can even remove humans from the loop and have the machine determine who's an undesirable or not.
How is this a serious question lol?
Uh? In what universe are openai and anthropic not shadow groups?
Oh, how nice of them, such charitable souls...
Doing what? Rushing to make trillion dollar companies, rather then developing the tech and their companies in a sustainable way? I don't even know what in the original comment this would be a response to.
The particular form of the pursuit of power practiced by OAI and Antrophic - the creation of trillion dollar companies, is one of the most benign in recorded history. It's so unnaturally benign, that I doubt it's going to survive for much longer.
Political power is when people act in concert, when leaders have people behind them because those people agree with how they're being lead etc., it's not something that happens unilaterally. You can physically force people, but for what when you can have robots that you don't have to force?
So that kind of leaves wiping out nearly all humans, for a few hundred thousand remaining people to get served by robots and play golf. Which I would say is what we're kinda on track for. But that wouldn't be a race between nations, that race would just be pretend to get the tech online; it would rather be a race between all humans and a bunch of sick people.
AI doesn't have to be used offensively to give a nation power. The machines can already design new software at the drop of the hat, it will at some point become capable of designing new consumer and industrial products. Putting a tireless intelligence at the task of looking for new optimizations in your industrial processes and factories will eventually give you economic dominance, one way or another.
Offensive uses don't need to be as extreme as you imagine them. We're already starting to see the effects of having meticulous automated hackers. Instead of a blitzkrieg, imagine your foes suffer from a constant stream of industrial accidents, unfortunate events and information leaks that can never quite be traced to your nation.
Finally, political power itself doesn't need to come from people acting in concert. There's always been wheeling, dealing, intrigue and bureaucracy. Once AI is sufficiently versed in legal matters, it'd allow political interest groups to overwhelm governments and private enterprises with objections and appeals, paralyzing their ability to carry out their agenda until they make concessions.
The possibilities of using intelligence for acquiring more power are literally endless..
But that doesn't mean that everyone has to use it.
We're only starting to see the beginning of the negative impact of AI to the job market and society. If if turns out that allowing most jobs to be replaced by AI is bad for society and the economy (super high unemployment - nobody buying more then necessities), then wouldn't it be smarter to not do that?
If that's the case, then each country can decide for themselves.
Dario's "Machines of loving grace" UBI/foodstamp "utopia" sure doesn't sound like a recipe for success.
The only leader I’m not aware of really taking a more active role in safety on the daily is Elon Musk, so I’d like to see evidence supporting what he is doing or not doing on that front if someone from SpaceX can share their experience. The evidence on the environmental front from SpaceX is very concerning and is disregarding the wellbeing of the world and local populations in my opinion.
Can someone who thinks this about Altman explain to me why he has repeatedly declined to take any equity in OpenAI? I genuinely want to understand your theory of mind here. Do you think he's actually in it for the power, and for some reason thought taking equity would undermine his effort at accumulating power? It just makes no sense to me.
No matter what practical benefits this technology might bring, if this runaway marketing campaign isn't stopped, in just a couple decades we'll see what the complete collapse of civil society looks like, a world where the institutions no longer see value in maintaining an educated democratic populous, and can relax back into the comfort of feudalism.
Or have you forgotten that Sam Altman was ousted by individuals that shared your concerns, and it was the OpenAI employees that rebelled against the board and wanted Altman back?
That HN post was infamous and nearly took down this site.
Value is simply the mutual understanding that something should cost a certain amount, and the system we've built emphasizes the need for profit above all else. When a set of companies comes along with a new, impressive technology claiming that they can automate away the white collar workforce, they're trying to capture as much value as they can.
It's not a moral sickness, it's market actors behaving rationally. Believing that people should be willing to leave money on the table because it's the right thing to do is a dead end, the entire system needs a rethink.
I don't agree that "I am just a market actor behaving rationally" somehow absolves you of any moral duty
I'm not an ethicist or anything but I have studied quite a lot of ethical frameworks and I've never encountered one that I think would agree with this
I didn’t expect a full throated defense of mugging and burglary on HN this morning, but here we are.
That's a touching sentiment or whatever but the nature of the thing is that it will inevitably be opened and that it cannot be closed thereafter. So (I think) we'd all be better off putting our efforts towards predicting the likely outcomes and figuring out how to make the best of them.
And if all the likely outcomes are very bad, then what?
You might as well ask "but what if all the options are bad" while standing on a sinking ship contemplating your next course of action.
I still don't know how Karpathy still works with those folks. I really think the guy is great but he makes me wonder
He would still be right and the field practioners would be wrong.
I do find capsules interesting though.
lifting people's standards of living -- is _not_ a technological problem. So I don't understand why people think that AI might solve this, even more so because AI is not primarily being developed or used to solve human or social problems, because there is little financial incentive to doing so (not much ROI in developing something that's cheap or free).
The Earth isn't lacking in resources, they're just distributed highly unevenly -- and correcting that distribution is a social and political problem, not a technological one. In fact, AI is more likely to exacerbate inequality than reduce it.
Let's say that AI allows us to crack the code to nuclear fusion. That would be amazing from a technological point of view. But the idea that this will lead to free, or even cheap, energy for all is rather naive -- all we have to do is look at whether the invention of efficient solar power (a limitless resource that is now relatively inexpensive to develop) has lowered energy costs for the average American [0]
[0] https://www.electricchoice.com/electricity-prices-by-state/h...
Like it would be fucking awesome if he could try find some more positive takes because it's tiring here a guy who basically got us into this mess, whine about how screwed we are.
Like good on him trying to raise awareness but I just find the guy so frustrating.
He doesn't search the internet to find an answer. "Search" results are nearly useless but the AI load does a good job.
He doesn't lean on AI to synthesize large information.
He doesn't use AI to build spreadsheets.
Etc.
Fortunately, he probably also doesn't use AI for therapy which two of the already crazy people in my family do.
Because a certification has to be fixed at a particular point in time. Korea's certification exams start with the 23 GoF patterns and go on to rote memorization of countless patterns and layered terminology.
The problem is that the way you learn for this certification is the old way, so someone who gets the certification ends up writing legacy code.
But if you always update the questions to the latest, then nobody can pass the certification. So, having seen how the certification system works in East Asia, I don't think it's all that good
No way to tell until after, though. So far, mutually assured destruction is working. It works until it doesn’t.
Same with AI. Thus far we’re better off with it than without it. That may change.
I mean, look what we did with fossil fuels. Some people are even still willingly burning them, even though the data is clear that our use of them has already started the total destruction of our civilization.
Is there a name for this, a technology that is so insanely useful that there is no way we won’t exploit it, but where the worst case scenario is total collapse, and impossible to determine well in advance?
(Fossil fuels I suppose don’t count as even the energy companies knew unambiguously decades ago that large scale use would eventually bake the entire agricultural system out of existence.)
The reason for that was more that the 3-4 great empires involved in the previous two were exhausted, Japan was defeated, China was in no shape, and US vs USSR had no reason to push for one, since the Cold War era gave them plenty of breathing room, roughly half the world each.
Both the USSR and US fought multiple wars directly against the proxies of the other. They weren't simply hanging back and letting others do the dirty work. In most any other era one of those would have become a full blown war between the two. Both sides pushed the line of not directly involved and then both sides looked the other way versus escalating. Nukes made the alternative too much of a risk for both sides.
They did, but didn't directly go for war between them or raised it to a 3rd world war, which is the point.
>In most any other era one of those would have become a full blown war between the two.
Would it? French and British empires were quite content not to go to war between them after Napoleon, when they felt they had their hands full with their own colonies each. After 1815 they've didn't do it, and even were on the same side in wwi and wwii.
Rather, after WW2 the idea of millions of people dying in a war became anathema -- countries would simply revolt against the idea.
Also, an estimated 20-40 million people have died in wars since WW2 (depending on how you do the counting), so it's not like wars have stopped.
The FDA analogy is structurally wrong for AI. Drugs have a fixed chemical composition and a testable biological mechanism; the FDA approves a specific molecule for a specific indication. A frontier model is a weight file that can be fine-tuned, quantized, distilled, or used with different prompts and toolchains after release. Pre-release testing cannot anticipate every post-release adaptation. The thalidomide comparison also misses that thalidomide's harm was a single, repeatable teratogenic effect. AI harms are emergent from use patterns that don't exist at test time. This is the core tension that makes any pre-release approval regime unworkable without also controlling how models are used after deployment.
That's a relatively silly question, don't you think?
>So one reason he might lament the status quo is because it has people thinking we have achieved AI when we haven't.
I don't think the article allows us to infer this at all.
>But also, it wasn't apparent until recently that even if the tech did work as promised, it will be an awful thing because humans will use it to bad ends. I don't think it's particularly blame-worthy that he didn't foresee that 50 years ago.
No, really, that would be extremely naive, we can expect better from scientists.
... no shit Sherlock
TFA could have been two sentences:
He warns that AI might cause serious problems: power failures, job losses, and greater social inequality.
He believes AI might improve our lives, but governments must make sure it is safe.