YHK - yet another hacker news clone

This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!

    give it a normal text prompt, and it
    continues that prompt by searching
    for the byte sequences that compress
    best.

One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.

So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.

The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?

Not without attention or something approximating it.

The fact that gzip is relatively fast should be your first clue that something important is missing.

Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.

If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.

Previous discussions of that specific page https://news.ycombinator.com/item?id=48557691