44 points coffeemug 1 hour ago 6 comments
jesdo 1 hour ago | parent
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
ivanjermakov 32 minutes ago | parent
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
jesdo 20 minutes ago | parent
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
sroussey 9 minutes ago | parent
Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling.
https://github.com/huggingface/transformers.js/blob/main/pac...