YHK - yet another hacker news clone

Quadrillion-parameter transformers and how to train them

22 points frozenseven 5 days ago 9 comments

frozenseven 5 days ago | parent

Paper: https://arxiv.org/abs/2610.37842

joefourier 20 minutes ago | parent

> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.

I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.

spwa4 19 minutes ago | parent

If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?

dingaling911 16 minutes ago | parent

DNA is probably the seed for the random initialization.

wernam 5 minutes ago | parent

It's nice to throw around claims like that, or like "microtubules connect the brain the quantum realm" (Penrose). Fortunately that's not how science works.

dingaling911 3 minutes ago | parent

Science starts with a hypothesis. If DNA has everything encoded, then it stands to reason that it influences the initial state of the brain.

derac 10 minutes ago | parent

The data is massively compressed and pruned. Where are the input tokens in an LLM?

derac 8 minutes ago | parent

That 200M figure needs much more justification as well, humans recieve huge amounts of data through their senses, particularly vision and touch.

notlibrary 2 minutes ago | parent

The planet has 20 quadrillions of ants