YHK - yet another hacker news clone

> Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.

I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.

Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.

That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.

+1. There aren't an infinite amount of serious security vulnerabilities out there.

> > The NHS approach: close source everything.

> Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.

Me too. The agents are much faster if it has access to source. NHS would have been easy hunting grounds. Now at least they don't stand out.

"a system should be secure, even if everything about the system—except the key—is public knowledge"

Should indeed, but we've all been caught with our pants down.

>It isn't realistic to expect new security vulnerabilities to be found at this rate

I wouldn't challenge fate like this.

You just need to use AI to test your AI code.

(What a wonderful new world we live in)

Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.

Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...

Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell

_Is_ AI generated code notoriously terrible at security?

My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.

Are we comparing to old models generating code? Or mythos-class models?

"New normal"? Is that phrase back again? It was heavily pushed in 2020.

> The rate at which we're churning out AI-generated code is going to make this worse

Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.

Rule #2 - do not blindly trust Ai output

https://susam.net/inverse-laws-of-robotics.html

> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.

I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.

I think that’s a fallacy, see https://news.ycombinator.com/item?id=50207540

The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.

1. The average human software-generator sucks at security. 2. LLMs are trained on human-generated software. 3. Therefore, LLMs suck at security.

Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.

Alas, our brave new LLM world is making iterating one billion releases per week all too easy.

And on the positive side, LLMs make remediation much easier too. It took me around 15 minutes and a few dimes worth of DeepSeek tokens to upgrade some gnarly, janky, exploitable PHP 7.x code to the latest fully patched version of 8.3 today. Without LLMs, I would probably have been having a serious discussion about whether we should ditch the app entirely instead.

> Most software, once mature, is mostly done.

The caveat being that most software never matures. Maybe it's a cynical take, but virtually all software I use regularly is updated at least once a month. Right now, I'm using Chrome, and I see the "Relaunch to update" button waiting for me to press it. I see the arrow on the Ollama icon telling me I have to relaunch it to apply updates. I have Visual Studio Code running next to my browser, and two extensions need to be restarted to apply updates.

But at least somebody is maintaining that software.

LiveOverflow has exploited the PHP vulnerability, it’s even linked in the article.

> especially due to persistent attempts to use regexes to validate untrusted input

This is not the case here, it's an untrusted regex which can be the issue.