Course overview: Understanding LMs#
This course deals with language models (LMs), in particular (but not exclusively so) on transformer-based language models like GPT-x or LLama. It covers topics that will equip participants with a better conceptual and practical understanding of what LMs are, how they work, and how to understand them. In the course, we will look (among others) at LM architectures, training & fine-tuning, prompting, mechanistic interpretability of LMs, LM agents and different evaluation methods. Participants will be offered both a technical perspective and encouraged to critically think about important topics relevant to cognitive science and society in the context of LMs.
Intended audience#
The course is intended for master students or advanced bachelor students (of, e.g., (computational) linguistics, cognitive science, or computer science) who are interested in language models / NLP / linguistics / AI / interdisciplinary approaches to state-of-the-art technical advances. Prior knowledge of LMs is not required; prior experience with programming in Python is highly encouraged.
Course formalia#
In SS 2025, the course consists of a weekly lecture (Tue, 9-12) and a weekly practical seminar (Thu, 14-16) which will systematically introduce and cover concepts from the lecture hands-on, in practical worksheets, where participants will work with (small) LMs themselves. This webbbook will host both lecture materials and practical materials.
The course is intended for 6 ECTS. There will be compulsory homework assignments and a final exam. Optionally, for 9 ECTS, students will conduct group projects.
Schedule#
The lecture overview below is preliminary and subject to changes.
Introduction
Neural networks and small language models
Transformers
Large language models, prompting & fine-tuning
Probing & attribution
Evaluation
Mechanistic interpretability
Further materials#
There are various courses on deep learning, NLP and LMs out there. To name a few, these are excellent sources for further related materials:
Large Language Models by Ryan Cotterell’s lab
Large Language Models by Percy Liang’s lab
Pragmatic NLG with neural language models by Michael Franke
Andrej Karpathy has an excellent 1h overview video.
We also highly recommend Brian Christian’s book The Alignment Problem for a more general perspective on AI and computational CogSci.
Previous iterations of the course#
Slides from the 2024 iteration of the course can be found here.