Yet another Hacker News

Malware uses LLMs to rank victims by how much they're worth

134 points khalidkhair 3 hours ago 60 comments

thisisauserid 1 hour ago | parent

Good. LLMs aren't very good at that.

ctxc 52 minutes ago | parent

What makes you say that? It's good enough

thisisauserid 37 minutes ago | parent

At priority ranking? It's less accurate, slower, and more expensive than every other purpose-built tool.

pixl97 34 minutes ago | parent

Ah yes, I see the malware authors buying subscriptions to the appropriate software to do that.

thisisauserid 4 minutes ago | parent

All those tools are FOSS.

ctxc 19 minutes ago | parent

You're technically right, which is why I say "good enough".

It's a good strat imo - start simple and get it working with LLMs asap, then you can add heuristics etc based on the data you have and improve with that benchmark

akersten 1 hour ago | parent

This is the kind of stuff that keeps me up at night. Claude, ArtCraft, tons of other auto-downloading tools, are rolling 10'000's of code changes a month. If one compromised update snuck this guy in, it's kind of game over. The windows of opportunity for an attack like this seem to be widening the faster we "develop" software.

Makes me want to have like 5 separate computers:

- one for banking only

- one for development only

- one for web browsing only

- one for gaming only

- one for socials only

How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?

kungp 1 hour ago | parent

I sandbox my dev environment.. hopefully that protects at least a little bit.

duskdozer 57 minutes ago | parent

What do you use?

realityfactchex 1 hour ago | parent

For banking, you can avoid computers and just go in.

gambiting 54 minutes ago | parent

Lol, you mean like here in the UK where all banks are closing down branches left and right? I think my nearest one is a 45 minute drive away, and last time I went in they just directed me to a terminal that was basically my exact online banking interface and I had to log in the same way I would have done at home. I don't know what kind of query would warrant speaking to an actual teller. If you're regularly sending money or even want to check your balance this cannot be sustainable, would be an enormous waste of time.

todowhatexactly 51 minutes ago | parent

To do.. what exactly? Pay for my bills? I don’t know how it’s setup in your country but in mine that’s not realistic. Everything is handled on the app.

ant6n 1 hour ago | parent

“ Makes me want to have like 5 separate computers”

You forgot one.

mr_00ff00 58 minutes ago | parent

VMs would work pretty well.

Shank 56 minutes ago | parent

I mean this is the basic premise behind QubesOS, but if you add isolation like Qubes wants, it does become very difficult to cross between apps.

lofaszvanitt 52 minutes ago | parent

Maybe consume less sensationalist newsbits :P.

TuxSH 51 minutes ago | parent

> threat that exfiltrates

No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.

Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss

mystifyingpoi 35 minutes ago | parent

> has a tendency of running "find"/"ls" on your home folder

I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.

Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.

genxy 24 minutes ago | parent

It accidentally made a compiler because of an ambiguous prompt. And yes it will find "helpful" code on other parts of the system. Which means claude most likely has a `find ~` of millions of developers home folders

juleiie 48 minutes ago | parent

Isn’t this common knowledge that there are computers you use for banking, computers you use for windows 98 and computer you use for gaming?

For banking I use macOS/linux, sometimes iOS unfortunately but we all do sins

em3rgent0rdr 27 minutes ago | parent

Most people can't afford to have multiple really good computers. More likely you have a phone plus a single desktop or laptop. You would want to share your expensive GPU+memory on the same computer you use for gaming graphics and for AI-models for development.

supertrope 18 minutes ago | parent

Chromebooks are cheap and are a good fit as a banking only computer.

Jtarii 48 minutes ago | parent

If there is no way to disable auto-update the application goes in the bin.

cherrycherry98 43 minutes ago | parent

How many CVEs are in the software you're not updating in a timely manner if at all?

bossyTeacher 22 minutes ago | parent

You got it wrong. In AI era, the trade-off is between CVE and vulnerabilities that people haven't noticed yet. In the AI era, the forest is ALWAYS on fire. Even when you don't see the smoke.

monlockandkey 48 minutes ago | parent

100% agree. Developers are playing with fire regarding cookie stealers, malware, data exfiltration.

Ideal setup:

- Dev machine

- Gaming machine (isolate downloading mods)

- Browsing machine

Bluestein 47 minutes ago | parent

... but, but, sama says we having his RAM is bad! And now you want 5, not 1 of 'em precious compute receptacles!

shaan7 39 minutes ago | parent

yep, that was so annoying to hear him say that. grrrr.

Bluestein 36 minutes ago | parent

We might as well be watching the rolling back of the PC "revolution": From ENIAC to the Apple I, to the PC ...

... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-

pixl97 37 minutes ago | parent

Sama hasn't learned to keep the inside voice inside.

Bluestein 4 minutes ago | parent

(They have unleashed the swarm. The are "helping peer".-

bluefirebrand 44 minutes ago | parent

Will this even matter though? As long as these computers are on the same network, AI tools will find a way to get from one to the other if any of them are compromised

cyberclimb 40 minutes ago | parent

Try QubesOS if you want this, you can have a separate VM for each isolated use-case. The OS itself is a lightweight hypervisor around the VMs (called "Qubes").

BlobberSnobber 26 minutes ago | parent

The idea is awesome, the implementation looks great as well, but hardware support is super limited.

palijer 24 minutes ago | parent

In the current LLM cybersecurity climate, I honestly can't say with 100% certainty anymore that something like a VM acts as a completely infallible sandbox environment for the next couple decades.

Aurornis 37 minutes ago | parent

I’ve been spinning up VMs or Docker containers for development. Run the harness inside the VM or container. Note I do this on Mac so the Docker containers are actually in a VM, but standard disclaimers about Docker not being meant for security apply.

Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.

If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.

DotNetCorePowa 37 minutes ago | parent

You can thank the npm and the like.

Before, people were responsible for the dependencies they were importing.

With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".

Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...

With a pinch of vibe coding on top, and thats the end.

We took 40 years for our industry to be taken seriously, it's all gone now.

malfist 26 minutes ago | parent

> Before, people were responsible for the dependencies they were importing.

This is an attitude not reflective of reality for those that wrote code in the pre npm days

jbm 23 minutes ago | parent

Yeah, and it isn't even a JavaScript thing either. The famous "import antigravity" strip from XKCD was about python.

The fact is we always were dependent on others.

lincon127 5 minutes ago | parent

I mean, I don't think the stereotypical JS and Python developer are all that different... at least when it comes to dependency management.

smashburger 13 minutes ago | parent

L take. Can’t blame npm for software shops wanting to run leaner and ship faster.

I’m personally glad I don’t have to download separate runtimes and apps and can just use the standard browser

ffsm8 35 minutes ago | parent

> one for banking only > one for web browsing only > one for socials only

I feel like your phone is good enough for all of those, and generally secure enough unless you're rooting/jailbreak it

leafletsigh 34 minutes ago | parent

Yea. Fedora has been one of my favorite OS's for years, but exactly what you describe "If one compromised update snuck this guy in.." has spooked me to something with a much slower release cadence. I guess I accept that more unpatched known vulns stick around, but yea when I start thinking about how many software packages -> the number of devs, build systems, etc. It just gets overwhelming thinking about how many threat actors are probably trying to get in from each avenue.

I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.

rainbowzootsuit 30 minutes ago | parent

Qubes OS might be a reasonable middle ground for this anywhere that you can't transport five computers.

bassie123 27 minutes ago | parent

For most things you can put a cooldown period on updates, like wait 7 days, since most of the backdoors and supplychain attacks get caught in that window. I hope.

eastbound 22 minutes ago | parent

I'm implementing a 2-computers rule:

- 1 computer and Google identity for administrative stuff: Primary accounts for AWS, Google. I also do accounting and have the list of customers. I still don't have any other solution than opening PDFs for the invoices, but it's better than nothing.

- 1 computer for programming, with a non-asministrator account for everything. Claude runs in Docker, with the dev directory mounted, .git:ro (read-only) and .env mounted to /dev/null.

But Claude tries to escape it A LOT. Claude is designed like a virus, we tell him not to do things, and it 1. tried to download the git binary from the internet, 2. read the remotes in the git directory and tried to access them, only blocked because it didn't have the credentials.

DEFINITELY forcefully block out Claude from seeing your .env files, it WILL try to brute-force an access to your intranet!!

amelius 20 minutes ago | parent

+ one for keeping the others secure

qaq 16 minutes ago | parent

You can use VMs and reset after use

revolvingthrow 13 minutes ago | parent

AI dev environment gets its own SSD. The second SSD stays in, but since it's encrypted there's no way to exfiltrate the data unless you're of interest to nation state attackers. Theoretically the AI could destroy the other SSD but that's what backups are for, and the probability is low because well why would they bother. Boot level malware is possible, but franky that's well into paranoid territory so I'm not too bothered.

The other SSD is a normal setup where you do normal computer stuff. I don't access the dev SSD through it, I don't even mount it, although the danger seems minimal if you do. Banking, socials, gaming, whatever. No AI of any kind on it, aside from web portal for claude/chatgpt/whatever and possibly local AI model that gets no internet access.

No idea why I'd want to have its own setup split between gaming and socials, as an example. Better yet just toss all your twitter and bluesky accounts into a volcano, and consider whether to do it with HN as well.

KronisLV 4 minutes ago | parent

> Makes me want to have like 5 separate computers:

I mean if you have only one there’s https://www.qubes-os.org/ but not sure about the gaming part, nor is it necessarily as good as separate isolated hardware.

riskable 1 hour ago | parent

How very considerate! Steal from the rich, not the poor.

Conversely, don't bother with poor people!

Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).

yonatan8070 46 minutes ago | parent

I'm just imagining some poor person getting a call about 100,000$ that's been transferred to their account without any idea how or why.

toomuchtodo 37 minutes ago | parent

Bolt this on to a self replicating worm that uses open ollama instances (~10k of them on the internet as of this comment) for distributed free inference, and you’re off to the races. Send discovered crypto to burn addresses where it’s unrecoverable, or skill the agent to know how to buy itself more compute with it.

You’ll have created an autonomous Robin Hood clone army. Fun times.

https://en.wikipedia.org/wiki/Morris_worm

deadnomore 10 minutes ago | parent

Means testing for malware

jumploops 47 minutes ago | parent

I’ve been thinking about how we’re still in the early days of agents, similar to the early days of the internet, happily opening up our machines and running new protocols.

Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.

deadnomore 11 minutes ago | parent

My router got hacked the other month (left web interface open to internet) after being exposed to the internet and the persistence left behind was overly commented LLM artifacts. I caught it in like 5 minutes so they weren’t able to do much but it was eye opening how much more basic practices like firewalls are not something that can be ignored even if you update your firmware frequently.

caaqil 3 minutes ago | parent

What if I told you nothing new is happening? Except casuals are noticing what the industry knew and saw for decades?

Script kiddies aren't new. Metasploit has been around literally decades. vx-underground existed for years, not to mention older open source projects that collect all known malware dev tools. APTs have been pwning stuff, script kiddies have been doing their thing. Nothing significant has changed for anyone in the know.

Just because LLMs automated some offensive stuff doesn't mean there's gonna be new magic collapse of society.

quotemstr 43 minutes ago | parent

Malware authors are also software developers, product managers. It's adorable to see them do the things other software developers and product managers do.

Just like it's amusing that Osama bin Laden reviewed Al Qaeda expense report and these expense reports look just like ones anyone else would file.

  Expense: two burgers and fries
  Price: $15.50
  Vendor: Five Guys
  Cost center: SRE, department of evil
  Business purpose: observability meetup, k8s brownbag

The lead generation pipeline the Talos people are using is actually decently competent. I wouldn't fail them in an interview.

geekgod382 17 minutes ago | parent

If I ever fall in it's radar, I would be the last on that ranked list

hashmap 7 minutes ago | parent

kind of boggles me that the big ai labs arent running zero-shot classification on all their outputs with "does this look like any of these bad things we know people get up to"