Yet another Hacker News

Amazon saves millions by replacing Sonnet 4.6 with Strands Decider

135 points aloukissas 1 hour ago 34 comments

oofbey 1 hour ago | parent

If their model costs 98% less than Sonnet, and Amazon is saving “millions” then these folks are raking in 2% of millions. Congrats I guess?

vincnetas 1 hour ago | parent

2% of millions ... so far. Take in to account growth potential.

Den_VR 1 hour ago | parent

small, open source, decision model optimized for fast experimentation, local development, and innovation...

But just think of the street cred!

mcintyre1994 37 minutes ago | parent

I think you’re missing that Strands Agents are open source models released by AWS, not an external startup that Amazon is paying.

swiftcoder 33 minutes ago | parent

Yes, this is more of less an advert, using their internal adoption as a case study

rrr_oh_man 26 minutes ago | parent

I yearn for the day of going back to make v. buy, not buy v. buy

sznio 1 hour ago | parent

why didn't they upgrade to further sonnet versions?

mikae1 59 minutes ago | parent

How about at least reading a couple of paragraphs from the post this thread is about before commenting? :)

Zanfa 33 minutes ago | parent

If they’re running on AWS Bedrock, lack of batch API support for more recent Anthropic models would be my guess.

coolgoose 1 hour ago | parent

Yea right... Paying market price of 3 usd per mil tokens on sonnet 4.6...

armcat 1 hour ago | parent

About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.

[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927

NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.

floro 51 minutes ago | parent

This is just the AI grift and uneducated hypebros. There is zero reason to use an LLM as a classifier. But AIbros with zero technical knowledge will push for it anyways even if it is 1000x slower and 1000x more expensive to run.

wodenokoto 46 minutes ago | parent

You can can't build an SVM-classifier for a ticketing system without domain data. You can ask an LLM to do it, and it will do it very well.

With the LLM you don't even need a data scientist to train the model, you just need someone who can connect to APIs. And the LLM is fine with adding and removing categories. No retraining.

It makes the initial faces incredibly fast. Then it is slow and expensive to run, so once the system settles, it becomes the target for model optimization, e.g, a dedicated, or semi-dedicated model.

IanCal 16 minutes ago | parent

There are absolutely good reasons to use smarter things for classification. Svm with tdf idf for example is super fast but will make very bad classifications at times in ways entirely unfixable with that system (thinking back to bats being an acronym for a theology degree classification or similar)

NitpickLawyer 4 minutes ago | parent

This is a bit too radical and misses some nuance. In order to start building classifiers you at least need some ML domain knowledge, like what to google at least. Meanwhile, point anyone at the jev/open-jev-clones API and have them start classifying in <10 minutes. There's some value there. (same used to be valid for LLMs before jev became a thing. You can use llms as zero-shot classifiers with the right API wrapper, w/ no domain data).

benrutter 44 minutes ago | parent

Yeah, its pretty wild! I think one of the benefits, but also potential risks, with LLMs, is you can just point them at anything without a lot of expertise.

Building an SVM isn't a tricky job, but you probably need a data scientist involved, and to think carefully about a labeled dataset.

I think we're seeing the modelling equivalent of something like electron apps. Where they waste a tonne of resources, but make an easier developement experience.

Will be interesting to see how much or little this changes with LLM costs.

jval43 36 minutes ago | parent

SVMs in big data were already well liked back in 2010 as they could be scaled via map-reduce.

An issue in industry was that companies would not allow training on their data across customers. Understandably so, as SVM dimensions are readily interpretable given some thought. General purpose datasets were ill-suited too depending on the task.

So you'd start from scratch every time, annotate features like crazy, and often didn't have enough training data overall.

LLMs "solved" that by just ignoring the previous rules and training on everything, which lets the necessary world knowledge seep back in.

I'm sure we'll see many combinations of LLMs and regular machine intelligence in the near future.

chaostheory 22 minutes ago | parent

The tech industry is not that far off from the fashion industry

autuni 18 minutes ago | parent

I think this kind of change will pop up more often.

In software engineering, we learned a long time ago that giant monoliths are not the answer. You want smaller, but specialized units that work together. They perform better at their specific task, are more maintainable, you can just upgrade one part of the system without having to impact the others, ....

I don't know where the focus on building a single model that can do everything comes from.

Geezus_42 7 minutes ago | parent

I was thinking about this recently as well. Smaller, specialized models that can run on device just make so much more sense to me than these huge generalized models that need huge infrastructure run.

phillennium 1 hour ago | parent

It's funny how AI sometimes feels like it went full circle:

— "Let's build efficient classifiers"

— "Forget about the silly classifiers, LET'S BUILD A MACHINE GOD"

— "Ah, that machine god is often too expensive and underwhelming. Let's build efficient classifiers"

Schiendelman 29 minutes ago | parent

The key is that the circle takes a few weeks instead of a decade.

Kaedon 1 hour ago | parent

At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?

I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.

f0cus10 49 minutes ago | parent

yes, there's a ticket for EVERYTHING. people have tickets cut in their names even decades after they've left

vasco 49 minutes ago | parent

Well if they use VPN point to point connections managed by AWS they'll get an email per week until they die for each VPN instance from AWS that they need to put in trash so I understand them.

sjdh76 12 minutes ago | parent

Well there is the UFO/Bigfoot/Poltergeist crowd.

If you dont fully pay attention to their theories about what they think is the reason for the suspicious stain on the ground in the parking lot, the queue is going to get flooded. If your Boss belongs to that club oh man god say the division.

tdhz77 53 minutes ago | parent

You can do this even more efficiently with the rust crate bonsai. You will have more control and you can create sceneries of logic. You can use incredibly cheap models this way and get way better results.

I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.

It’s shocking how effective this is.

soon_to_be_sb 27 minutes ago | parent

Where can I read more about this!?

lencastre 39 minutes ago | parent

counting the miliseconds until the insurance industry picks this up

3eb7988a1663 39 minutes ago | parent

Classifying into one of 20 groups? Did anyone try scikit learn?

lofties 8 minutes ago | parent

I was going to say, and I say this a little bit too often, a BERT model can do this even faster, with even less memory...

nlpnerd 21 minutes ago | parent

Founder of Seldon AI (www.seldon-ai.com) here. We are working on dev tools to help developers with this kind of refactoring and continual production work.

Basically there is a whole host of tarpit usage of LLMs in application backends today. This is simply another case of LLMs enabling people to build what they cannot productionize or maintain.

Today developers without any AI/ML skillset can prototype a fancy new feature with LLMs, what used to require collecting and curating data to train a pipeline of ML models then serving them in production.

The problem comes when it's time to take the new feature into production. That $20 cost with prototype/beta workload is now $30k per month in LLM costs, and those hallucinations that never happened on their 200 test samples is happening 600 times per day.

They are then stuck, because they lack the knowledge and budget to build and run the production ML pipeline properly, the infra stack necessary to handle production workload. Management also doesn't understand why they have to invest in rebuilding something that "seems to work" with "outdated methods".

These system one or decision models are a step up from LLMs when it comes to this usage. Problem is that they are still difficult to work with (limited in expression power) for many application logic. Despite the initial popularity, I hear about a lot of frictions refactoring LLM based backend with them, and multiple cases of production rollbacks to LLMs already.

I think system one models is a sensible paradigm, and we are on the precipice of something great, but there is a lot more work needed above the model layer to make this work.

hubertzhang 4 minutes ago | parent

People use machine learning models before and after LLM is invented.