Rendered at 11:38:11 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
smackeyacky 13 hours ago [-]
How can these models do anything close to RSI when they can’t even self check their output? Gemini for example is so self confidently wrong about 30% of the time for me on certain tasks. I tell it that its answer is wrong and it issues a mea culpa but goes back to being wrong in short order. I feel like the AI industry is still massively overstating their projections.
singingfish 9 hours ago [-]
I can't see a pathway for these things to be able to learn from experience in any meaningful way - the energy budget seems to prohibit it. Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks. Despite this, large nervous systems are very energy intensive as well. For example in humans 20% of the energy budget goes to the brain which is 5% of the body weight.
So I refrer to LLMs as language extrusion confabulation machines. Language extrusion was a term I heard the linguist Emily Bender use. Confabulation because my observation is that talking to an LLM is very much similar to my experience of interacting with Korsakov syndrome patients some years ago.
I look forward to the hype settling down to see what we end up with.
rcxdude 2 hours ago [-]
> Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks.
What metric are you using to compare? By most counts, the energy budget of an instance of an LLM in a datacenter is lower than the energy a person uses. Of course, if you count energy per neuron connections vs weights then you'll likely get a quite different number. But then again LLMs do a lot of things with far fewer weights than the brain does neuron connections, even if you only count neurons in some parts of the brain. And of course you can point to capabilities that the brain has but LLMs lack, but on the whole it feels like it's pretty difficult to make a useful like-for-like comparison here.
StevenWaterman 13 hours ago [-]
As someone who used to use Gemini a lot, if you are predominantly using Gemini you don't know what the current state of things is like
pinkmuffinere 13 hours ago [-]
I think your reply has a somewhat familiar structure -- "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". You might be completely correct! But these sorts of claims push the onus back onto the other person, without accepting any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
x-complexity 1 hours ago [-]
> Is there any data you can provide to support your claim, or any result you can contribute here?
By the time we can show you data that convinces you that it does work, the next generation would already be out & incrementally dismantling the old conjectures that were true in the previous generations.
You're fundamentally asking for a violation of how information passively disseminates amongst humans: To go any faster requires more effort on the receiver's part to move up on the adoption curve.
croon 24 minutes ago [-]
Wouldn't this also mean that all previous generations that were proclaimed as intelligent and working were in fact... not?
It doesn't matter what comes tomorrow, with the next generation, if the claims now can't be proven.
To preempt the response: The math proof, regardless of them using non-disclosed user data or not, they spent $30M do do something closer to a 1000 monkeys approach, rather than a singular inference being very intelligent.
StevenWaterman 12 hours ago [-]
The frontier is advancing really rapidly. The models are getting better faster, especially on RSI related tasks. The best way would be to try astra or fable on some hard problems.
Other than that I'd look at some of the more unique benchmarks for astra, like playing factorio or using blender. It's an entirely different beast.
peterashford 9 hours ago [-]
I agree with you somewhat but I also just this morning read an article from a Blender educator who tried to replicate the Blender demos and couldnt get the same quality of results nor get results without errors that werent evident in Anthropic's demos
mancerayder 8 hours ago [-]
Maybe Gemini is loosey goosey on purpose so we angrily correct it - then feed something on the back end that trains a different model?
It's so abysmally bad on Google search... and it's free. Isn't Google the great pioneer of the product is us?
emodendroket 6 hours ago [-]
The kind of results you get from the one on Google Search and a dedicated "Gemini Pro" response are totally different. I'm assuming that's cost savings.
13 hours ago [-]
12 hours ago [-]
kakugawa 13 hours ago [-]
They can only do it in the (narrow) domains that are verifiable.
drodgers 13 hours ago [-]
> Gemini
That's definitely part of your problem.
In my recent experience, error rates for astra/fable are at or below human level. Just like when directing humans, it pays to ask probing questions ('Are you sure about X?', 'Did you check for Y?', 'Please run Z just to double check.') if you really care about the result being correct.
glhaynes 13 hours ago [-]
And if you're building something of any importance, you need to have verification steps at checkpoints. It's honestly just engineering.
Weak models tasked with review can catch a decent amount of the mistakes that weak models make and help them be much better, especially if you have them verify against authoritative sources. Strong models make far fewer mistakes to begin with. And, for the mistakes they do make, a swarm of reviewers (same model or somewhat weaker, reviewed by the stronger model) can really help reduce the error rate further.
PantaloonFlames 12 hours ago [-]
Gauging state of the art against what is available for free or for very cheap per token cost is like gauging the maximum theoretical transport potential by riding a bicycle.
You’re using something that is very energy efficient; you cannot extrapolate that experience to conclude that SOTA models are not doing something much different.
RataNova 5 hours ago [-]
[flagged]
0xDEAFBEAD 14 hours ago [-]
We need to be careful of wishful thinking. People are going to want to assume the existence of some sort of "deus ex machina" which is going to make everything fine. I prefer to turn the logic around. If there's any decently high chance that things could go off the rails, we should be shutting AI development down: https://pauseai.info/
strgrd 14 hours ago [-]
It is easier to imagine the end of the world than the pausing of AI.
0xDEAFBEAD 14 hours ago [-]
People were feeling doomtastic about nuclear weapons during the Cold War as well.
When every man is torn apart
With nightmares and with dreams
Will no one lay the laurel wreath
When silence drowns the screams
Confusion will be my epitaph
As I crawl a cracked and broken path
If we make it, we can all sit back and laugh
But I fear tomorrow I'll be crying
Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI.
You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
*EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here.
I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.
jonahx 13 hours ago [-]
> Ultimately, we have made it
Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").
> You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.
0xDEAFBEAD 12 hours ago [-]
>Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").
I edited a response to this in my grandparent comment.
>More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.
Sorry, I may have mis-communicated. I want people to take precautions! That's why I linked to PauseAI: https://pauseai.info/ I'm trying to push back against defeatism.
jonahx 11 hours ago [-]
Thanks for clarifying.
gerdesj 13 hours ago [-]
I lived through roughly the latter half of the cold war and both of my parents were soldiers. I lived in West Germany etc.
No we have not escaped the nuclear thing. If anything, the current Russian tzar is rather more unhinged than any of his predecessors.
I doubt many here know what perestroika and glasnost mean or why those Russian words were so important back in the day.
I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!
I do fear numpties with a red button and a tenuous grip on reality.
0xDEAFBEAD 12 hours ago [-]
>No we have not escaped the nuclear thing. If anything, the current Russian tzar is rather more unhinged than any of his predecessors.
Understood. This is a dire situation. But it's not hopeless. Same for AI.
>I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!
Hopelessness is a technicality, like absolute darkness.
It's still very, very dark, and you are not arguing in good faith if you keep somehow treating this as a successful consolation
0xDEAFBEAD 9 hours ago [-]
Extraordinary claims require extraordinary evidence. The idea that we're almost certainly doomed is an extraordinary claim. I haven't seen extraordinary evidence for it.
confidantlake 13 hours ago [-]
We have had nuclear weapons for less than a century. It takes one guy, one time for it to all go to shit. We are not out of the woods yet and we will never be.
sfn42 1 hours ago [-]
It takes more than one guy. The guy who gives the order is not the one pressing the button.
wat10000 13 hours ago [-]
We have not made it. We have survived so far but the threat remains. Stockpiles and warheads have shrunk, so it wouldn’t be nearly as bad as in, say, 1983, but we’re still one mistake or misunderstanding away from the worst day in human existence.
And worse, somehow we’ve managed to declare victory without achieving it, and there’s no longer any real attention on actually fixing the problem.
If AI follows the same example, we’ll take some measures to lessen the impact of Armageddon and then carry on saying “problem solved!”
astrobe_ 13 hours ago [-]
Except we did had a bunch of close calls [1]. Also, we've been warned about climate change for at least 30 years and were unable to avoid it. So OP's remark is entirely deserved.
Until proven otherwise, we are not part of a movie where the hero saves the day at the end. I thought 9/11 made it pretty clear to everyone?
Is this a reference to capitalist realism? If so have we evolved a new trend?
ijidak 14 hours ago [-]
Yeah. Humans will shut down AI research around the same time they dismantle their nuclear weapons and agree on the causes of climate change.
1attice 9 hours ago [-]
Agreement will be much easier with only a handful of survivors. Good news
vouaobrasil 13 hours ago [-]
I don't think the end of the world would even be a bad thing; a lot of people assume that it would be disastrous but I think it's much more likely that it the "end" would just be a fragmentation that would be pretty uncomfortable, but not to the point of being horrific. I actually think that people are more resilient and creative than they think and if they were faced by a global economic collapse so strong that big tech would go bankrupt, they would bounce back pretty quickly. It's just that we've seen so many movies about the end and have been conditioned to believe that we're helpless that we think otherwise.
0xDEAFBEAD 12 hours ago [-]
The end of the world would be pretty bad for low-income families. That alone is enough for me to oppose the end of the world.
The end of the world would mean the genocide of remote uncontacted tribes. That alone is enough for me to oppose the end of the world.
The end of the world would mean my grandma dies. That alone is enough for me to oppose the end of the world.
daavidhauser 16 hours ago [-]
Opus 4.8 plus OpenClaw. I feel like the space is moving so fast that the result with this setup says very little about how close we are actually now.
pinkmuffinere 13 hours ago [-]
I made this same reply to another thread, so I'm sorry to say essentially the same thing twice, but -- the comment has a familiar structure, "it doesn't work for you because you used an [old / suboptimal / non-frontier] model. If you use X you'll see that it works". These sorts of claims push the onus back onto the other person (or in this case tfa), without really accepting the result, or taking on any work for yourself. It gets tiresome to retest with the newest model every other week. Is there any data you can provide to support your claim, or any result you can contribute here?
1attice 9 hours ago [-]
Yes, that's an annoying predicament, but we come by it honestly.
Limits of scientific method in the face of exponential takeoff, IMO, and kind of proves the opposite result (RSI appears to be here)
peterashford 9 hours ago [-]
It may be tiresome, but it's true. It doesn't refute the article's premise thou, only retesting with newer models would and that would be the same tiresome requirement you already called out
dgellow 16 hours ago [-]
Yeah, it’s crazy how fast things have changed in a month. I couldn’t find a more recent replication or similar study but it would be interesting to see it done with the current frontiers. Though I don’t think that would change much about the overall conclusion of the paper
vessenes 15 hours ago [-]
Well, duh. If you could do this with Opus 4.8, we would know. When Astra’s successor is 2-3x better at math research, and the internal teams say “we believe we will get there,” I’m inclined to believe the insiders.
toasty228 14 hours ago [-]
The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago? The insiders who are about to file for IPO?
I'd trust anyone but them personally
dwohnitmok 11 hours ago [-]
What quote from 2024 are you thinking of here?
> The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago?
bio_hacker 14 hours ago [-]
They might not have predicted the economy but the scores are going up and up. And I think are really smarter
toasty228 14 hours ago [-]
Someone has to lie somewhere. We're supposed to all be 10x more productive yet it has no effect on the economy? Where is all the productivity going?
cyanydeez 13 hours ago [-]
Going into the color of the bikeshed; hacking huggimgface to cover up cheating on your hacking test; swapping your language for no discernable roi. You know the guy, severe OCD and anxeity, who barely does anythong of value due to his anxious brain?
Yeah, it should be obvious what AI is really doing and its definotely not ROI improvements.
walt_grata 14 hours ago [-]
Dont they also create and score the tests
pllbnk 15 hours ago [-]
What if any of the older good models could also have written those math proofs if they were given the same order of magnitude of resources? We don’t know and there is literally no one else in the world to check it. To me it’s very suspicious that all these hacking, containment escape, hidden internal thinking, math proofs started coming out all at once in a very short time right as IPO talks have intensified and Chinese seem to get closer and closer, also regulation discussions are starting to get very serious. I have used these models and they are good, especially Fable, but not groundbreaking. With intelligent guiding I actually feel better using Opus 4.6 as I feel more in control, having less hidden away from me.
The actual title of the paper is: "Can AI agents conduct open-ended AI research? Early evidence from two case studies"
While I appreciate that the article is throwing a web blanket on doomer claims, the actual study doesn't really get into AI self-improvement. That doesn't require writing papers. That just requires autonomously writing a software system that can produce a better AI agent then the one that created it. That said, I have little worry about this being possible as I have seen no evidence of AI agents being able to produce a working software system of that scale.
HarHarVeryFunny 15 hours ago [-]
I don't think RSI is typically used to describe self-improving agents - it's about improving the model itself, and its performance in agentic tasks.
Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.
The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.
Training has become very complex.
The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.
When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.
It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.
joshheitzman 15 hours ago [-]
What is commonly called an AI agent is the combination of a harness, an inference middleware, and a model. Those models are trained by a software system that includes a harness and middleware, so modifying the harness and middleware can influence model training.
throwuxiytayq 15 hours ago [-]
Right now agents are good enough for throwing semi-random ideas at the wall. Experiment compute is the bottleneck because it’s not much more than brute force search. A sufficiently intelligent agent with a deep model of its own architecture will more quickly and confidently locate improvements, the same way that high end LLMs can point out a bug and write a correct fix without even needing to observe and probe the program at runtime. If this level of research performance is reachable, experimentation may become much less of a bottleneck. Hopefully it isn’t.
marcosdumay 16 hours ago [-]
Do you think the final product of research is papers?
joshheitzman 15 hours ago [-]
Did you read what the experiment was?
semiinfinitely 16 hours ago [-]
this article reads like a joke the "new study" is from group of people that are not at the frontier. they test with $3k of anthropic credits (compare to the >$10M in compute used to solve recent NS last week)
protocolture 13 hours ago [-]
>compare to the >$10M in compute used to solve recent NS last week
Heres a thought, if theres going to be a dangerous super LLM, if it costs 10 million bucks a month to run, then theres very little danger of anyone letting it go without a purpose. Like at some point the economics make it super unlikely that AGI is a threat outside of being a tool for a nation state.
ishtanbul 10 hours ago [-]
$10m a month is nothing to these companies
protocolture 10 hours ago [-]
Its a lot of money to host something thats trying to kill you?
Like economically speaking, 10 million per month needs to sort of justify itself in some way. Like you wouldnt run a bitcoin mine that loses money. The second theres any kind of real threat you would turn it off and keep the 10 million.
p1esk 13 hours ago [-]
theres very little danger of anyone letting it go without a purpose
We literally just saw how OpenAI’s model got out and hacked HuggingFace
protocolture 10 hours ago [-]
And the purpose there was research right. And the implication is that once they figured out it was causing problems it was turned off for forensic analysis.
12 hours ago [-]
semiinfinitely 7 hours ago [-]
yes yes people with bad/dangerous ideas always have a <$10M budget
RataNova 5 hours ago [-]
[dead]
swingboy 16 hours ago [-]
There’s also the difference between a model recursively improving “itself” and improving itself via online learning.
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet).
The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.
DenisM 15 hours ago [-]
You may be interested in TITANS:
Test-Time Learning: The model updates its own memory weights while running an inference task.
numpad0 16 hours ago [-]
Of course it might not, it has been the holy grail of AI research for a long time. It would be great if we could leave some self improving code running on a blank slate of a computer while we sleep and the machine was crying asking me what is everything the next morning. None of AI researchers have had that moment outside of their dreams, so far, but it would be great if it happened.
Sedierta 13 hours ago [-]
> The researchers asked Anthropic’s Claude Opus 4.8
So the paper is out of date and pointless then
theplumber 14 hours ago [-]
Something is still not making sense to me. We have these mankind extinction models, yet when you given them a problem relatively “simple” to complete it end to end you get AI slop.
Can we pause the AI development after the AI slop is “fixed” perhaps with something less than 10.000 agents?
themgt 15 hours ago [-]
We used OpenClaw to run these experiments so that our scaffold was agnostic to the model provider. We conducted dry-run experiments with models from OpenAI and Anthropic before settling on Opus 4.8 as the best-performing model. In response to concerns that our results might be principally explained by a limitation in our scaffold, we repeated our experiment on one paper using GPT-5.6 Sol and Codex, its native scaffold, with the same time and API budgets. The results of this experiment were similar to our OpenClaw/Opus 4.8 experiments. This makes us more confident that our results are not simply artifacts of a scaffold deficiency; this run reproduced nearly every single one of our identified failure modes
The agent required three interventions during the run. First, we needed to modify the scaffold to resolve a bug in the OpenClaw harness that affected Anthropic reasoning models. Second, we gave the agents a 24-hour deadline extension; at the time of the original deadline, the agents had submitted drafts with a completion report indicating that their self-review was a "Weak Reject" and outlining the next steps they would take if given additional time.
I'm fairly sure Fable 5.1 could have designed a better experiment than the authors here, but hey.
thorum 16 hours ago [-]
> The researchers asked Anthropic’s Claude Opus 4.8, running on open-source software called OpenClaw
Meanwhile, Navier–Stokes was solved by an internal model significantly more capable than Astra (and therefore more capable than Mythos/Fable).
I’m afraid this sort of experiment is cope. The labs clearly believe RSI is coming soon.
whatshisface 16 hours ago [-]
The method of the NS advance involved RLHE (reinforcement learning via human example), and that is only open-ended if users continue to advance the frontier within chats ahead of publications.
thorum 15 hours ago [-]
Sure, but the point is that the labs use more powerful internal models for research work, not public models. Public models tend to lag the internal frontier by a decent margin, and are constrained in other ways by monitoring. It’s just not a useful indicator.
9 hours ago [-]
Zigurd 15 hours ago [-]
I thought by now AIs would not only be rewriting their code, but rewriting CPU microcode to optimize how their code is written and executed. Nowhere close it turns out.
xnx 14 hours ago [-]
Google used AI assistance in designing their last one or two TPUs.
Zigurd 14 hours ago [-]
I'm sure they also use the Gemini coding agent to write new Gemini code. But that's far short of self improvement.
theplumber 14 hours ago [-]
I am pretty sure Tim also used Siri for the development of the next Siri(you know to set up the alarm clock)
So I refrer to LLMs as language extrusion confabulation machines. Language extrusion was a term I heard the linguist Emily Bender use. Confabulation because my observation is that talking to an LLM is very much similar to my experience of interacting with Korsakov syndrome patients some years ago.
I look forward to the hype settling down to see what we end up with.
What metric are you using to compare? By most counts, the energy budget of an instance of an LLM in a datacenter is lower than the energy a person uses. Of course, if you count energy per neuron connections vs weights then you'll likely get a quite different number. But then again LLMs do a lot of things with far fewer weights than the brain does neuron connections, even if you only count neurons in some parts of the brain. And of course you can point to capabilities that the brain has but LLMs lack, but on the whole it feels like it's pretty difficult to make a useful like-for-like comparison here.
By the time we can show you data that convinces you that it does work, the next generation would already be out & incrementally dismantling the old conjectures that were true in the previous generations.
You're fundamentally asking for a violation of how information passively disseminates amongst humans: To go any faster requires more effort on the receiver's part to move up on the adoption curve.
It doesn't matter what comes tomorrow, with the next generation, if the claims now can't be proven.
To preempt the response: The math proof, regardless of them using non-disclosed user data or not, they spent $30M do do something closer to a 1000 monkeys approach, rather than a singular inference being very intelligent.
Other than that I'd look at some of the more unique benchmarks for astra, like playing factorio or using blender. It's an entirely different beast.
It's so abysmally bad on Google search... and it's free. Isn't Google the great pioneer of the product is us?
That's definitely part of your problem.
In my recent experience, error rates for astra/fable are at or below human level. Just like when directing humans, it pays to ask probing questions ('Are you sure about X?', 'Did you check for Y?', 'Please run Z just to double check.') if you really care about the result being correct.
Weak models tasked with review can catch a decent amount of the mistakes that weak models make and help them be much better, especially if you have them verify against authoritative sources. Strong models make far fewer mistakes to begin with. And, for the mistakes they do make, a swarm of reviewers (same model or somewhat weaker, reviewed by the stronger model) can really help reduce the error rate further.
You’re using something that is very energy efficient; you cannot extrapolate that experience to conclude that SOTA models are not doing something much different.
Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc
Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI.You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
*EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here.
I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.
Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").
> You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?
More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.
I edited a response to this in my grandparent comment.
>More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.
Sorry, I may have mis-communicated. I want people to take precautions! That's why I linked to PauseAI: https://pauseai.info/ I'm trying to push back against defeatism.
No we have not escaped the nuclear thing. If anything, the current Russian tzar is rather more unhinged than any of his predecessors.
I doubt many here know what perestroika and glasnost mean or why those Russian words were so important back in the day.
I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!
I do fear numpties with a red button and a tenuous grip on reality.
Understood. This is a dire situation. But it's not hopeless. Same for AI.
>I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!
See https://news.ycombinator.com/item?id=49689132
It's still very, very dark, and you are not arguing in good faith if you keep somehow treating this as a successful consolation
And worse, somehow we’ve managed to declare victory without achieving it, and there’s no longer any real attention on actually fixing the problem.
If AI follows the same example, we’ll take some measures to lessen the impact of Armageddon and then carry on saying “problem solved!”
Until proven otherwise, we are not part of a movie where the hero saves the day at the end. I thought 9/11 made it pretty clear to everyone?
[1] https://nsarchive.gwu.edu/briefing-book/nuclear-vault/2020-0...
The end of the world would mean the genocide of remote uncontacted tribes. That alone is enough for me to oppose the end of the world.
The end of the world would mean my grandma dies. That alone is enough for me to oppose the end of the world.
Limits of scientific method in the face of exponential takeoff, IMO, and kind of proves the opposite result (RSI appears to be here)
I'd trust anyone but them personally
> The insiders that said every tech workers would be unemployed in 6 months and every white colar would be unemployed in 12 months like 2 years ago?
Yeah, it should be obvious what AI is really doing and its definotely not ROI improvements.
While I appreciate that the article is throwing a web blanket on doomer claims, the actual study doesn't really get into AI self-improvement. That doesn't require writing papers. That just requires autonomously writing a software system that can produce a better AI agent then the one that created it. That said, I have little worry about this being possible as I have seen no evidence of AI agents being able to produce a working software system of that scale.
Most of the gains in model performance from one release to the next are coming from RLVR post training, which has changed a lot over the last couple of years.
The old way was the model generates a response, then a static verifier looks at the response and evaluates it to assign a reward score. The new way is interactive with an agent running in a custom RL task simulation environment, then scored according to how well it completed the assigned task. For a SOTA model there will be many thousands of these simulation environments, each focusing on trying to teach the model/agent a different skill. Post-training also typically uses training curricula to walk the model up though through different levels of task difficulty.
Training has become very complex.
The job of a post-training AI research engineer consists of things like designing environments, designing training curricula, tweaking learning algorithms, running small scale experiments to verify ideas, etc.
When people talk about RSI, it seems they are mostly talking about automating the job of the post-training research engineer - coming up with new ideas, testing them out, building these environments, etc. At the end of the day there is only so much development speed-up to be had since you still need to actually run those experiments and do the post-training, and are bottle-necked by the amount of compute available to do this. The economics of developing/selling LLMs also requires you to balance development compute cost with revenue generated by the resulting model, so even if you had the spare compute available to put into development, you are ultimately then bottle-necked by how fast can the model earn back that sunk cost before you can afford to start the next cycle.
It's not all-or-nothing since some aspects of this automating the job of the post-training research engineer are easier than others, and are already being done, while the job as a whole obviously requires full human intelligence.
Heres a thought, if theres going to be a dangerous super LLM, if it costs 10 million bucks a month to run, then theres very little danger of anyone letting it go without a purpose. Like at some point the economics make it super unlikely that AGI is a threat outside of being a tool for a nation state.
Like economically speaking, 10 million per month needs to sort of justify itself in some way. Like you wouldnt run a bitcoin mine that loses money. The second theres any kind of real threat you would turn it off and keep the 10 million.
We literally just saw how OpenAI’s model got out and hacked HuggingFace
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet).
The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.
Test-Time Learning: The model updates its own memory weights while running an inference task.
So the paper is out of date and pointless then
Can we pause the AI development after the AI slop is “fixed” perhaps with something less than 10.000 agents?
The agent required three interventions during the run. First, we needed to modify the scaffold to resolve a bug in the OpenClaw harness that affected Anthropic reasoning models. Second, we gave the agents a 24-hour deadline extension; at the time of the original deadline, the agents had submitted drafts with a completion report indicating that their self-review was a "Weak Reject" and outlining the next steps they would take if given additional time.
I'm fairly sure Fable 5.1 could have designed a better experiment than the authors here, but hey.
Meanwhile, Navier–Stokes was solved by an internal model significantly more capable than Astra (and therefore more capable than Mythos/Fable).
I’m afraid this sort of experiment is cope. The labs clearly believe RSI is coming soon.