21 points 0o_MrPatrick_o0 1 hour ago 5 comments
SyneRyder 10 minutes ago | parent
TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.