Yet another Hacker News

Show HN: Flide: Make Jev-like decision models up to 9.5x faster

27 points unikcc 5 hours ago 5 comments

Decision models need their own inference engine.

So we developed Flide, a high-performance and exact inference engine for Jev-like decision models.

Flide achieves up to a 9.5x speedup over vLLM and SGLang.

It now supports 30+ popular local Jev-like decision models.

Start to make your decision immediately:

pip install flide

unikcc 4 hours ago | parent

Hi HN! I'm one of the Flide team.

We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.

Here is the source code: https://github.com/OmniJev/Flide

Look forward to hearing your feedback and ideas!

rytill 1 hour ago | parent

Under what specific circumstances is it actually 9.5x faster?

Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?

How did you achieve the biggest speed up?

Are there any tradeoffs? “While keeping exactness” implies not.

Are there any specifics about how one must structure their context to take full advantage of it?

tancop 47 minutes ago | parent

I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything.

The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.

jarym 49 minutes ago | parent

I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!