26 points ankitvgupta 1 hour ago 6 comments

datadrivenangel 33 minutes ago | parent

Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.

datadrivenangel 33 minutes ago | parent

it only cost them $10,000 in Claude API usage. :)

hmate9 28 minutes ago | parent

You get that much usage in a monthly $200 subscription.

dofm 7 minutes ago | parent

And a kitten.

risyachka 17 minutes ago | parent

Fpga is software-driven, model can verify its output easily.

Zero surprise it did it

codekansas 11 minutes ago | parent

This is cool

Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...