made this the day after liltransformer. instead of one sentence it reads a whole text file, you pick how many words it looks at, and it keeps predicting words for as long as you want. probably.
its basically the same model just a bit bigger, 16 numbers per word instead of 8.
problem is it works with whole words. so "cat" and "cats" are completely different things to it, and it literally cant say a word that isnt in the file. thats why real models use tokenizers, which is what i did next.