YHK - yet another hacker news clone

If you don't run your own LLM proxy, how do you know whats going on?

21 points amrrs 4 hours ago 12 comments

voodooEntity 3 hours ago | parent

Well that was one of the reasons why i build my own agent/harness - to have actual full observability of what it is doing and why it is doing what it is doing.

Don't really understand why other agents to give those insights but well found my solution.

skeptrune 53 minutes ago | parent

most interesting link i've seen on this site in a while

hoppp 47 minutes ago | parent

Yup. Setting up liteLLMProxy on some servers as we speak and will start playing around.

I can't orchestrate agents in sandboxes without it.

1. It can issue virtual keys, so provider API keys never go near the sandbox

2. I can see everything

3. Guardrails

And much more

Looking forward to it.

DelightOne 26 minutes ago | parent

1) What kind of guardrails are you using? Do you do models like prompt injection checkers? Or what guardrails are useful that do not know the environment the agent lives in?

2) I heard Bifrost also works for this. Did you try that as well, pros/cons?

stavros 36 minutes ago | parent

Does anyone here use LiteLLM for this? How well does it do? Can I use it with my Claude subscription?

verdverm 23 minutes ago | parent

I use GoModel and its amazing. My favorite features are virtual model, the support for arbitrary request labels, and the nice dashboads / ui. Creator has more sophisticated report generation on the near term roadmap. He's super responsive to feedback and ships fast. (except for his recent trip to the US west coast to meet founders and investors, he's EU based)

You can use and control all.your subscriptions or aoikeys through a gateway like this, so you don't get pwned like what just happened with OpenCode (or an OpenRouter like vendor). GoModel has some ambitions like this, but is open source so you can run it on you own infra, with plans for self hosted enterprise as well (there are a few features already like SSO). We are rolling out the free version at work right now.

I run GoModel at home and have hyper detailed history of every agent interaction, across models running on my sparks and OpenCode sub. You can call me a committed superfan of this project.

impure 32 minutes ago | parent

OpenRouter does all this for you. It's pretty nice. Although for apps I use another server to lock down the requests even more and prevent the client from seeing my API key.

soared 26 minutes ago | parent

One man’s fun is another man’s “idk what any of these words mean, I pay $20/mo for some company to abstract this complexity away”

For clarity, I do like blog posts like these.

sroerick 8 minutes ago | parent

Seeing some good stuff in the comments, but how are people doing this?

jubilanti 6 minutes ago | parent

Yeah you should run an LLM proxy, but imagine a world where LiteLLM and all the other LLM shovels weren't buggy vibecoded messes. And with the velocity of updates I feel there has to be vulnerabilities there. Or am I just holding it wrong? Anyone got a good alternative?

NeutralWanted 3 minutes ago | parent

I'd love to do something like this, or use API keys, but I still don't understand how people justify the cost of using API keys vs. subscription.

I tested it once, I'm in on the 20x claude max ($200/month) and I either max my weekly limit every week or come really close. But it only took 2-3 days to hit $200 spend with api keys.