first ml thing i ever made lol. you give it a sentence, pick a word from it, and it tries to guess the next word.
the thing is it only ever trains on the sentence you give it, so its not really predicting anything, its just memorising. but it has all the actual transformer parts (embeddings, attention, the output layer, the loss, adam)
its super tiny. each word is 8 numbers, 2 attention heads, 1 layer, all in pytorch.
python3 mdl.py