ok this one is an actual language model. 8.2m params, llama style, trained from scratch on tinystories (a dataset of simple little stories made for tiny models like this).
i trained my own tokenizer this time (bpe, 8000 tokens), used the llama model from hugging face (4 layers, 256 wide, 4 heads, 384 tokens of context) and trained it on my macbooks gpu.
it writes little stories that mostly make sense and then kinda wander off. its also my first model on ollama so you can just run it:
ollama run navthings/lilstory
i also ended up using it for a paper, thats the next post :)