26 points ankitvgupta 1 hour ago 6 comments
datadrivenangel 33 minutes ago | parent
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
risyachka 17 minutes ago | parent
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
codekansas 11 minutes ago | parent
This is cool
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...