navthings https://navthings.github.io/ nav, machine learning engineer and researcher. notes on pretraining, finetuning and scaling experiments. en sprout can talk now https://navthings.github.io/blog/sprout.html https://navthings.github.io/blog/sprout.html Thu, 01 Oct 2026 05:00:00 +0000 sprout finished pretraining on sep 27. 523m params, 12 billion tokens from fineweb-edu, dclm, cosmopedia and project gutenberg, all on the free kaggle tpu over 4 sessions. this time i actually saved the loss history so theres a real loss curve. teaching lilbase to talk https://navthings.github.io/blog/lilchat.html https://navthings.github.io/blog/lilchat.html Sat, 26 Sep 2026 09:00:00 +0000 a base model just continues text. if you ask it a question itll probably write 5 more questions instead of answering. to make it a chat model you finetune it on a bunch of conversations so it learns the pattern: you say something, it answers, it stops. lilbase, 297m params on a free tpu https://navthings.github.io/blog/lilbase.html https://navthings.github.io/blog/lilbase.html Wed, 23 Sep 2026 09:00:00 +0000 everything before this was trained on my laptop, but i wanted to make a proper base model, like one thats actually read a decent chunk of the internet. my macbook air was never gonna do that. kaggle gives you free time on a tpu v5e-8 (8 of googles ai chips) so i rewrote the whole thing in jax and moved it there. tale, bedtime stories for my little brother https://navthings.github.io/blog/tale.html https://navthings.github.io/blog/tale.html Sat, 12 Sep 2026 09:00:00 +0000 my little brother kept asking me for bedtime stories so i made a model that tells them. i wrote a paper on how much data matters https://navthings.github.io/blog/dataset-size-paper.html https://navthings.github.io/blog/dataset-size-paper.html Sat, 05 Sep 2026 09:00:00 +0000 i wanted to see how much the amount of data actually matters if you keep everything else the same. so i trained lilstory 4 times on 1%, 5%, 10% and 15% of tinystories. same model, same settings, 5000 steps each. lilstory, my first real language model https://navthings.github.io/blog/lilstory.html https://navthings.github.io/blog/lilstory.html Fri, 04 Sep 2026 09:00:00 +0000 ok this one is an actual language model. 8.2m params, llama style, trained from scratch on tinystories (a dataset of simple little stories made for tiny models like this). bigtransformer, now it reads a whole file https://navthings.github.io/blog/bigtransformer.html https://navthings.github.io/blog/bigtransformer.html Mon, 31 Aug 2026 09:00:00 +0000 made this the day after liltransformer. instead of one sentence it reads a whole text file, you pick how many words it looks at, and it keeps predicting words for as long as you want. probably. my first transformer (it only knows one sentence) https://navthings.github.io/blog/liltransformer.html https://navthings.github.io/blog/liltransformer.html Sun, 30 Aug 2026 09:00:00 +0000 first ml thing i ever made lol. you give it a sentence, pick a word from it, and it tries to guess the next word.