LLMs Unplugged: Understand AI by building it yourself

Abstract

LLMs Unplugged is a series of hands-on activities demonstrating the fundamental ideas behind large language models. You start by manually counting word patterns in your training text (can be as simple as a kids book), recording these patterns either as tally marks on grid paper or even by cutting the paper into pieces and putting them into labelled buckets. You then generate new text (based on this training data ) with some dice rolling or pulling the paper pieces back out of the buckets. After doing this by hand, something clicks: you'll see that LLMs like Claude or ChatGPT work using the same basic principles, just at a vastly larger scale.