88 points armcat 1 hour ago 7 comments
vatsachak 1 hour ago | parent
I remember being presented Word2Vec as an undergrad in 2016 and I could not predict the sci-fi reality that is today
resters 1 hour ago | parent
The idea that a word is defined by the company it keeps was decades old. Mikolov’s insight was that you could learn it at scale with a very simple model, and that meaning would fall out as geometry. Obvious in hindsight, foundational in practice.
BOOSTERHIDROGEN 1 hour ago | parent
I still in awe how could someone understand well in math, is it pure nature or nurture. Could any average possible to grasp the beauty of math, starting from high school level math understanding.
samrus 1 hour ago | parent
Deserved. word2vec is a beautiful peice of science. The hypothesis behind it was so amazing. I kinda feel like everything else in LLMs (including the attention mechanism itself) is just engineering around the representation learning breakthrough that word2vec demonstrated
jacquesm 19 minutes ago | parent
And with applications far outside the original scope, which I think is one of the hallmarks of foundational discoveries. As soon as you have that one puzzle piece a whole lot of other stuff comes into focus.
_false 8 minutes ago | parent
This is my first time hearing about Tomas in a while. Does anyone know what he has been up to since word2vec?