When Genius Fails: The Intellectual Arrogance of the AI Labs
weightythoughts.comAfter he got fired, he wrote a 165 page thesis on AI in early 2024 that all ended up becoming true in 2026; you can read it for yourself and be the judge: https://situational-awareness.ai
That's what he based the fund on.
It's generational foresight not available to 99.999999999999999999% of 25 year olds. I want to get ahead of the default cynicism and "oh it's just dumb luck woe is me" aspect of HN.
This is why hedging and neurotic levels of risk-management are necessary. Running a highly levered and highly correlated portfolio is a disaster waiting to happen regardless of whether your overall thesis is correct or not.
There are not enough human 25 years old in history for this many 9s. You had 21 nines, which would only make sense if there were 10 billion billion humans ever and that he is the only one. However, the current estimate is there were 107 billion humans ever alive in history, so your statement is impossible even if the estimate is off by 10000 fold.
It's a rhetorical device that I have yet to see used to good effect. The kind of rhetorical device that undercuts the argument rather than strengthens it.
...no? The essay is literally titled "The Decade Ahead", its predictions have not come true in 2026.
> The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word
I'd argue even the 2025/26 part of that isn't true, though the "many" in "many college graduates" makes the claim very difficult to verify.
But even putting all that aside I think it's still a valuable evidence point when discussing "intellectual arrogance". No doubt this 25 year old is very smart when it comes to AI. But there's this belief that if someone is smart at one thing then they're surely smart at everything. Turns out "smart at AI" doesn't automatically translate to "can sensibly manage a $45bn fund".
He didn’t have the foresight to avoid collapsing his hedge fund in a minor selloff by hedging.
You might be overselling his ability to predict the future.
And I want to get ahead of this absurd romanticism. I have plenty of counterpoints in all types of technical domains (3D printing, blockchain, you name it). Oracles don't exist, just stories of right place right time survivors.
> That's what he based the fund on.
His thesis was so true that his fund folded. Fancy that.
Yes, being unable to pay margin call means your fund is folding.
It's not part of the strategy or operating agreement of a GP/LP structure to sell a specific fund to another investment firm at a discount. The fund may buy and sell assets to create returns for LPs but selling the fund itself at a discount is not a successful exit, e.g. they folded.
What happened to SA is not what anyone would call “a fund folding”. Normally you’d assume folding means the fund went bankrupt, or closed at a loss.
This fund is not closing and not on a loss. It’s not bankrupt, and apparently it is opening up new positions and continuing to invest in its main theme last I saw it in the news.
It is on a major drawdown from its peak a few months ago, but it’s still up for the year, most of its investors are likely well in the positive. If you make a huge investment and it goes up 5x before trimming its gains and ending up 100%, you’ll be annoyed, you’ll wish you sold at the top, but you’ve still doubled your money.
Getting margin called and closed this way is not the same as going bankrupt.
> Getting margin called and closed this way is not the same as going bankrupt.
Technically we don't actually know the details here since no one here is privy to the specific fund structure - which is very likely WAY more complex than reported.
So, realistically we're arguing semantics. I said it folded and for intents and purposes if you're getting margin called and have to sell, then that's folding.
Just like in poker I can fold and not play anymore and still have a stack of chips.
In fact, the returns were so astronomical that they took a $35B loss (around 67%) which means they were still UP 80% for the year in 2026.
Today they have around $10B under management.
As an investor, how would you feel about being up 80% for the year? Exactly. They're having a phenomenal year.
He currently still has $10B under management.
What's the point of posting blatantly wrong details? It's pretty easy to look them up.
What actually happened is that AI-assisted training of LLMs (mind you -- stuck on that same old transformer architecture, with nothing resembling a medium-term memory, and fine-tuning still doesn't really work as a form of "learning") is right now giving us ~linear-ish progress... beecause we already hit the slowdown in the S curve of what human researchers can achieve.
We are also energy-constrained in ways that the exponential forecasts have no answer for other. Compute is not getting more efficient fast enough to achieve anything close to exponential growth driven by growth in computing power. There are social, moral, and political limits on the amount of energy we can dedicate to AI training in any given time period. And it's not just energy, we're short on RAM wafers and water for cooling and eventually we're going to hit other limits on various minerals and elements of the supply chain etc. etc. So even if it were true that R&DEffort == RateOfProgress without resource constraints, we're likely not going to see that exponential progress because we also need corresponding exponential scifi-scale progress in computing efficiency, cooling, and, energy delivery.
Basically this kid made a bunch of very clever and grand predictions which were always kind of ridiculous and no, they have not come to pass in 2026 and we are not at all on track to achieve AGI in the next 12 months, despite what Altman keeps trying to make the public think.
"Any reasonable expected rate" does a lot of work here. For example, I haven't expected LLM contributions to open math problem that early.
For example, doing it at Columbia is meaningful. Doing it at a random school, not so much.
His strategy was to trade on MNPI available to him via relationships that no larger fund/bank compliance department would sign off on, in a portfolio no fund/bank risk department would allow.
Calling the fund “situational awareness” and then demonstrating a complete lack of situational awareness was rather amusing.
> It's generational foresight not available to 99.999999999999999999% of 25 year olds.
As a side note, it’s pretty insane how many people of the AI industry have a relationship to effective altruism
> Gebru and Torres argue that these ideologies should be treated as an "interconnected and overlapping" group with shared origins. They claim these constitute a movement that allows its proponents to use the threat of human extinction to justify expensive or detrimental projects and consider it pervasive in social and academic circles in Silicon Valley centered on artificial intelligence. As such, the acronym is sometimes used to criticize a perceived belief system associated with Big Tech.
https://xcancel.com/porterstansb/status/2083606266388029691
(I also enjoyed the meandering in to Amex and Wells & Fargo's founding stories)
(I mean sure from sibling comments I get that this is, in fact, not performance art but you gotta admit that if it were it'd be damn good!!!)
It's an interesting look into who gets opportunities in our society.
His "main thing" is success in calling economic impacts of undervalued large shifts, and the premise of the fund is basically that the same thing is occurring around AI, where he also has specific subject matter expertise.
I'm going to need better evidence to believe he has any skill.
From what I can see, his investors could have just bought AI related stocks themselves.
i know nothing about this specific person, but the typical rule is that if it has already reached the general news cycle, you are too late.
Aschenbrenner only created his Situational Awareness fund 2 years ago, so he wasn't exactly prescient in predicting the rise of AI - he just had enough conviction to go all in, with leverage, on an investing theme that was already pretty obvious.
They though they would laugh all the way to the bank...but that laugh was already the problem...
"We Need To Talk About Leopold" - https://youtu.be/rE75WvOtcu8
Regardless he was still able to build a giant pile of cash before alighting it on fire.
His fundamental thesis is as an AI maximalist, with expertise on who will be the winners.
His fund completely blew up, even though his fundamental thesis still may be correct.
He made a ton of money early, but then he just kept increasing his bet size via leverage. The problem is that these stocks may be going up, but they are EXTREMELY volatile. When the whole market went down with some of the Iran news, his margin levels dipped too low, and his lenders issued margin calls. He had to sell his stakes at losses, and his fund collapsed.
Now, part of the problem is he was not a legacy hedge fund with great relationships with his banks, because if he was, he might have been able to postpone the margin calls. Had he been able to do that, he would have been ok, because his positions actually recovered within a few weeks.
It didn’t matter, though, because he was forced to sell at the low point.
In other words, even if your long term bet is correct, if you are maxing out your margin to max out your bet, you are one volatile dip away from blowing up.
No matter how good the bet, you have to be able to ride out the volatility to survive.
It’s like if someone offered you a bet on a coin flip, where they will pay 5x your bet if you win. How much should you bet on each flip?
Well, by expected value you should bet everything you have, because your EV is 2.5x, so the more you bet the higher your expected value.
However, if you bet it all, you have a 50% chance of losing all your money and not being able to make any more bets.
But thats sort of the whole problem, believing you know a lot about the future of AI, even if you’re correct, does not mean you have the expertise to manage a fund around that belief, and it’s absolutely shocking to me he was able to get that much money
If you run it in unsafe mode, it won't prompt. The problem is that you would probably want to run it in a container and that's annoying.
Too true. This isn't limited to AI, either. The most obvious example in my lifetime was during peak blockchain hype, when people who had never worked in finance convinced themselves that blockchain was going to act as the backbone for how money gets moved around. As if the problem that needing solving was Bank of America doesn't trust Capital One to update a number in their database.
The nice thing about AI, at least, is I can always push back and tell people, "Sure, we can do this with AI. I just need you to use Claude or ChatGPT manually to prototype how it would work." This normally results in the requestor realizing that there's human judgment calls involved in the inputs, process, or outputs that require meatbag intelligence.
I specifically remember a prolific HN commenter working on an AI video startup claiming that by the end of 2025, children would be able to create Hollywood-quality, feature-length movies using available AI video tools.
I believe this person's startup has (wisely) pivoted to building AI-powered tools for the movie industry.
In other words, this is a failure of degree, not of category.
I don't know how long it will take, but Hollywood is obviously screwed the moment something like Stable Diffusion gets temporal coherency and stops randomly adding limbs, because the energy cost per image on my laptop multiplied by how many frames in a film and $0.1/kWh, is ballpark the same as the cost of a cinema ticket.
I still have no idea how far we are from something that would be good enough to render a Hollywood film. Well, a Hollywood that isn't a complete disaster.
Writing it is a separate problem, my experiments there show gradual improvement that has now reached the level of "obviously not publishable but also slightly less bad than going on reddit and sorting posts in short story groups by most recent instead of upvoted".
Turns out there is more to it than that.
But do also consider that when cameras became cheap, *painting* is what became somewhat less valuable, and cinema became possible in the first place.
And consider that when cinema got sound, live musicians in theatres went away:
The time is coming fast when the only living thing around a motion picture house will be the person who sells you your ticket. Everything else will be mechanical. Canned drama, canned music, canned vaudeville. We think the public will tire of mechanical music and will want the real thing. We are not against scientific development of any kind, but it must not come at the expense of art. We are not opposing industrial progress. We are not even opposing mechanical music except where it is used as a profiteering instrument for artistic debasement.
If the AI could write… it can, but the degree to which it can is not enough; if the AI could make coherent videos… it can, but the degree to which it can is not enough.
"If". It's a matter of degree, but functionally the answer *today* is "no", because today when some humans make a video with AI, they need to put in a lot of effort to throw out the rubbish attempts by the AI.
When it stops being "no", and not before, is when Hollywood is screwed.
I'm sorry, you're asking "why would Hollywood be screwed?" with regards to a tech that lets you write a short prompt and get an artefact out?
One where it already does this for stock photography, short scripts, and short clips?
Where random people already do this for fun because it's so cheap?
Where I'm talking specifically about a hypothetical which cost-anchors this for the price of a cinema ticket, i.e. $10 in creates a movie tailored to whatever the person prompting it happens to ask for?
You can't see how this has the same impact on Hollywood that sound recordings had on live music performances during cinema?
You can't see how anyone who wants a new film for some specific mood can just ask for that in this future, and skip the entirety of Hollywood?
I'm not using fiction as evidence here, so don't misunderstand it as such, but consider how the Holodeck is written in Star Trek: you talk to the computer and it gives you what you want, and if you don't like it half way through you pause and tell what to change. For a variety of obvious reasons, Trek's script writers and practical effects people didn't put in all the glitches we see in current GenAI models, but I'm asking you to imagine a future where GenAI models also don't make such mistakes.
> Movies with sound put theatre musicians and some actors with unfortunate voices out of a job, but Hollywood thrived.
"but Hollywood thrived." is Texan sharpshooter, it's picking the winner after the fact.
I specifically gave live theatre musicians as an example of the opposite, those who were screwed by the new tech.
Addendum: eg https://www.latimes.com/archives/la-xpm-1988-07-22-ca-7511-s... in California.
They want to watch movies other people watch with actors they know.
I also expect that of the 39.2% of the population, the cancer is diagnosed much later in life than it was for the 91 movie crew members, I doubt 41% of the crew was over 50 in 1956.
>...There was no excess risk of cancer mortality in southwestern Utah, for single or grouped sites, with the single exception of leukemia which showed statistically significant odds ratios of 1.45 based on 62 deaths at all ages, and 2.84 based on nine deaths at ages 0-14. The finding for childhood leukemia was based on different time periods and geographic comparisons from those of two earlier studies in which no such excess was found. Mortality from all cancer sites combined was significantly lower in southwestern Utah than in the remainder of the state, even after adjustment for the higher proportion of (lower risk) Mormons in southwestern Utah.
https://pubmed.ncbi.nlm.nih.gov/3788954/
Trying to figure out when this strange idea about The Conqueror first appeared, I saw references to a 1980 article in People magazine. People magazine is not actually a peer reviewed journal.
On that one, they were right enough that there are an enormous number of businesses that use crypto to avoid something government related.
Yeah but only because a meatbag is going to use the product :)
Are they expecting the equity they get in some AI company will save them personally from economic ruin, so they're fine with that outcome for everyone else?
Or are they expecting AI technology will magically make our society communist?
Yes. Basically based on my conversations with coworkers who went to these frontier AI labs: this is happening anyway, so they may as well make the money while they still can.
There's got to be a special name for that kind of treachery. I suppose "collaboration," but that doesn't have a nasty enough ring to it.
Like a fascist occupation, that dystopia isn't "happening anyway" without collaborators.
I just call these people "cowards"
Which is different from just being rich and chilling. Rich and chilling implies a stable society. But life isn't a video game where you hit some Absolute Level 99 and are impervious to everything. There's no outside justificatior for you - only you are. In this hypothetical society, when GrokBots rule the world, you think they'll care that you're a millionaire? You need to get into the inner circle.
I see some really smart professors/PhDs even, going to SV, starting startups, networking, etc. That is consistent with the worldview.
But someone who says that AI can replace them but just works at a lab, at least has the belief that they will be automated last. Maybe they believe they can become friends with the AI, I don't know. But let's stop and analyze the situation with a bit more nuance, for more than like, 5 seconds.
This is not the same thing as communism. You can still have private ownership, differing levels of wealth, entrepreneurship, and competition. It's just that having money is not necessary for survival or to lead a reasonably comfortable life.
And they don't believe it will magically happen -- it will take intentional action by governments to get there. This is, perhaps, the reason the AI labs are talking so much about AI replacing people's jobs. It's not arrogance, it's to make sure our leaders are thinking about this now instead of when it's too late to guide the outcome.
I thought they were talking about AI replacing people's jobs, because that's music to investors' ears.
Think about how much wealth concentration would happen if they could hit their goals eat that much of the economy.
IIRC, Elon Musk has literally compared the effect of AI on people to the effect of highway construction on an anthill (that's in the way), and kinda shrugged his shoulders.
Maybe they're right, maybe not, but it is ironic.
It seems like, at least for tasks like distilling the work done at lower levels and passing it up the management tree, LLMs should be good at that. Maybe they could take out some levels at least.
And they can be programmed to be 100% selfless and on-mission: no turf battles or other internal politics. Everybody’s job involves office politics of course, but my gut says management has a lot more of it than engineering, so zeroing that out could be more significant.
The models don’t generate what’s right and good: they generate what people are likely to generate.
I dunno. You are probably right. But it’d be an experiment. It wouldn’t be an experiment if it was a sure thing.
> They managed to fine-tune models to be good at math and coding, despite the vast majority of humanity being awful at those…
Think about the kind of people who are leading the companies making these models, and their underlings who will be leading the fine tuning.
Math and coding are fairly objective. Fine tuning for morality and ethics is going to reflect the biases of those people.
Trying to replace the foot-soldiers of capital in a capitalist system would be bold, if one could raise the capital for the experiment.
Attempting to replace this layer is unwise in this system. Someone else would need to take the blame, and you can be sure it won’t be the sociopaths at the top. It’d be the losers who get the blame AND do the work. At this point the work would slow, then stop. Or at least that’s how I see things playing out?
More interesting, to me, is what happens to the clueless in that scenario. Without work these folks would probably revolt?
> Too true. This isn't limited to AI, either.
That's so true there's a term for it: engineer's disease/engineer's syndrome.
https://en.wiktionary.org/wiki/engineer%27s_disease
https://ask.metafilter.com/297591/Origin-of-the-term-Enginee...
I have seen this personally, where merely identifying a possible solution (harmless, reversible) is misinterpreted as selecting and implementing that solution (may be irreversible).
This is how all people solve problems. Now everyone are not great at all parts, but this really is what anyone does.
Surely the title ”engineer” means something else?
Documenting and analyzing your solution is an extremely important part of managing risk, which is what engineers actually do.
If you've got engineer's disease, those "other fields" include things like psychology and politics. It's a combination of arrogance and myopia.
If your expertise is legitimately relevant and you know where your expertise ends, that's not a problem.
"You don't got to frame walls anymore!", something that is the most enjoyable part of residential construction, takes like a day of work at most, and can be done while on autopilot. "We can instantly generate wall plans for stud location from top-down blueprints!", oh that thing people do in their head because they just space it all on 16 or 24 inch centers and it doesn't actually matter which side you start on? "We can wirelessly drive your tractors!", the easiest task on the farm which all the weeks of maintenance and planning and setup led up to?
Now don't get me wrong, there are many things that can be improved in those industries certainly, but you aren't going to know what those things are until you actually go on the job site and learn how they currently do it daily. Most all of the low hanging fruits have already been picked one way or another and a fancy step stool is not going to be industry changing.
I am trying to fathom a more catastrophic failure in communications than the lab CEO's using this narrative. The reality ends up being closer to Radiology jobs [1], but we're never able to fully realize this because the press is running around like headless chickens and unloading the fear on a public that then reaches for any leverage they can get (see: Datacenter NIMBY-ism).
Any short term FOMO valuation bumps they got from this narrative has to now be liquidated into paying premium for server racks in a warehouse.
Reminder folks: There is no evidence that AI is not normal technology [2]
[1]: https://www.apollo.com/wealth/insights-news/insights/daily-s...
[2]: https://knightcolumbia.org/content/ai-as-normal-technology
Literally the Godfather of AI did [1]. The rest is pablum, Radiologists do far more than just protect against liability. I hope you never find out why they are so critical to vast number of patients getting the worst news of their lifetime. As OP lays out, life and its professions is just not as simple as these narratives would like us to think, and we keep backfilling reasons for why all dooming and glooming consistently comes up wrong. Just selective memory writ large to keep the literal infinite scroll of doom going.
[1]: https://www.auntminnieeurope.com/imaging-informatics/advance...
I was talking about number of scans analyzed, not time. In fact, the decrease in the amount of time it takes to do radiology work is exactly why they are now paid more. Everything in american medical care is about profit, hospitals love this new paradigm because they get to charge the same amount and do 5x as much work per radiologist. Since incentives between patients and providers arent aligned we've entered this hellscape where providers are constantly ordering radiology work that is completely unnecessary because the MBAs are pushing high margin stuff to the entire organization. I personally have many stories from my 2 doctor friends where the business forces them to overprescribe in the stupidest ways. It sucks that the whole thing is a racket and that Hinton couldnt see that but I dont think that makes him wrong about the long term future of the profession.
Hinton said 10 years, what do you say? Could you maybe take from his prediction being wrong that you should practice some humility? Of all the pro-AI arguments people make, the "eventually it will happen" one is one of the strangest. Yes, and eventually the sun will blow up, so what the fuck are we talking about here? Most smart people ten years ago thought truck drivers, cab drivers, radiologists, and customer support would be obsolete by now, hasn't happened.
Where Hinton was wrong: He figured if each radiologist could do twice as much work we'd only need half as many and therefore there would be an oversupply of them. Instead what happened is we now do three times as much radiology. I went over the economic reasons for this in the other comment.
I dont think its correct to say Hinton was wrong on his core claim that AI will radically change radiology, he was just wrong about what that looked like.
It's obvious that deep learning did/would radically change the part of any job that required analysis of patterns or images. The core point of the OP is exactly the claim you're making, which is that you seem to think that all a radiologist does is either look at images or "sign off" on what AI has done after looking at the pixels.
As I understood it, the attraction of blockchain was more that the money could get moved around without Bank of America and Capital One knowing about it, and therefore without those financial institutions reporting the transactions to the government as they are required to by law. Which of course screams "money laundering" and "tax evasion" to the government. Yes, there are possible legitimate reasons for wanting to be able to move cash around anonymously and invisibly, but they're basically rounding error compared to the ones mentioned above.
In other words, blockchain on this view was a classic example of the old saying that you can't apply a technical solution to a social (or in this case political) problem.
> Climbing Mount Everest and getting a PhD are both hard. I wouldn’t say that doing one means you can do the other “no problem.”
and here:
> Surgeons and electrical engineers may be extremely smart and accomplished, but that doesn’t mean they have the ability to weigh in on climate science.
Feels tangentially like a version of Nobel disease[0], discussed here[1] a couple of weeks ago.
---
https://www.theguardian.com/us-news/2026/aug/13/trump-aircra...
This exactly. As someone who once lived in Silicon Valley and then ended up interviewing 700+ CxO's of profitable (or nearly profitable) SaaS businesses in various domain specific fields, it amazes me how many prominent "experts" in SV believed they were consistently "right" simply because they made enough bets to eventually get one right. Maybe not in enterprise value terms, but in volume of companies there are FAR more successful and profitable SaaS businesses that you've never heard of because they were started by entrepreneurs in their respective domains (the MD who started an EHR platform or the truck driver who started a TMS solution, etc.).
I just skimmed it, so I'm hoping someone with more fortitude can summarize.
May I ask, did you get anything out of it? I felt I wasted my time reading it, but that's what makes me think I'm missing something.
Was social media a clean win for human social interaction? Was fast food the replacement of food that was promised in the 70s? Were doctors and shrinks in phone apps rather than real life a clean win for diagnosis, understanding and improvement of life?
But society was completely shaped by the market in these questions anyways in a way that is impossible to deny.
While the author might be right about the arrogant PhDs imposing their view on other fields they know jack shit about, it doesn't make it less true when you have billions backing it and making it a reality, it will eventually be it, even if it's arguably worse.
It's a word that's thrown around a lot (particularly in tech) that rarely has much to back it up. Often, it seems to be used for 'confident young man', or even 'grifter', and then when the idol falls, the same people quietly focus on lionising someone new.
Interesting.
Remember that HN downranks posts with certain ratios of comments to upvotes, or with a certain level of activity - I'm not sure exactly how it works - to prevent flamewars. Whether this is a good measure is up to debate, but all articles get the same treatment if they hit these thresholds.
anti-ai posts get downvoted by pro-ai people/bots. pro-ai posts get downvoted by anti-ai people/bots. (i suspect one of these teams has more bots than the other, but that's just a guess). in any case, the world keeps spinning.
and there can be several motivations for it (e.g. downvoting a competition's posts, someone upset with how much pro-ai stuff there is on their favorite subreddit, etc.)
And agree that manual downvotes (or comments) are not always in good faith either (though I'd argue they're intrinsically still better than bot votes, but not as if you claimed differently :)
A bit of that is to be expected from any disrupter, but there’s now serious questions about if these startups have the basic competencies required to survive long term.
I know people preaching about LLMs being the Next Industrial Revolution, who couldn’t place the industrial revolution in the correct century.
I have people at work talking about jobs and technologies getting replaced by AI who don’t understand how a lawn mower works, who don’t know who Lenin was or what a differential equation is, because they’ve simply been writing JavaScipt since high school.
There's idiocy and overconfidence everywhere. When more money is involved, I guess it's more entertaining to watch. At least they're trying something. Maybe there's more effective ways to fleece people of their money that yield greater societal benefits, but I think there's some beauty in someone thinking they have it all figured out being able to lose their money on a shitty bet.
Now once they cry for government help, that's where I care.
Having a machine that's even as smart as most human's + cheap robotics, does seem to lead to us having no jobs in near-ish future eventually. (5-15 years)
Though obviously AI progress could hit a brick wall tomorrow.
It’s more important to keep an open mind, and readjust your assumptions regularly.
This sort of audacity is a large part of the reason people increasingly hate AI
Hedge funds often get 20% of the upside. Pocket it, and then when it goes bust just walk away with the money. Defend yourself by saying "everyone went bust that year."