YHK - yet another hacker news clone

Yeah. The article is not misleading. The title is, but you know, headlines are all lies that's why you read the article.

If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.

What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.

> who read the original post

I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that

At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.

So you mean that the "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading? You should post that.

Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.

As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.

They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.

They did, except they also did write an abstract and a title that will mislead anyone who isn't going to read the full paper in details.

It's fair to say that this is malpractice.

Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.

I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.

If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.

The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.

Here's the work that shows how rule-based logic can be compiled into transformer weights:

* Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981

* Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 (code at https://github.com/deepmind/tracr)

So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.

Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.

That’s what I’m excited for: the raft of posts over the next few days as the inspired set to work

Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced

I was just watching a video about this. They had this survey question for AI researchers asking when they think AI would take their job. The reason they asked this was that would be the point where AI starts recursive self improvement. If it’s able to replace the job of an AI researcher, then it could keep improving itself.

Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).

It's obvious that recursive self-improvement has been happening and accelerating quickly for some time.

AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.

Isn’t that what the Google dreaming thing is?

According to this person, some problems are only "solved" by modern models: https://nitter.netbub.com/AlexKrentsel/status/21173766837246...

(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)

I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.

They are also biased to see from what institution you come from..

Someone has to already be doing this right?

You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.

You do realize how academia works, right? Granted, it is often dysfunctional, but this ain't that.

Trying to find every single angle and option to break something is how we verify the truth of something.

If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.

This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.

How do you prune out the 'fails to prune out behaviors' behavior? If my model is failing to prune it out, can another model prune out that behavior?

> how you could prune harmful capabilities from models and build a more efficient training system

In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.

If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do

"[...] how you could prune harmful capabilities from models [...]"

We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models

"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"

If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.

It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.

If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.

And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.

It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.

Agree, it makes sense, a lot of people are catching on to LLMs for reverse engineering finally, but its been possible. LLMs can and will eventually flawlessly dissect models. I experiment with Claude, I ask it tasks I know are possible, but have no idea the solution, it can reverse engineer obfuscated projects faster than most talented people I know in that field (talking about games now not AI). I heard a developer say their latest release would take at least a month to figure out the obfuscation and encryption, Opus 5 did it in like... 15 minutes of talking to me (I did throw it the right questions / suggestions). I do that all in the name of seeing just exactly what is possible.

Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.