Course overview: Understanding LLMs#
The course “Understanding LLMs”, taught at Uni Tübingen, focuses on Large Language Models (LLMs) like GPT-x or LLama. It covers topics that will equip participants with a better conceptual and practical understanding of what LLMs are, how they work, and how to understand them. In the course, we will look (among others) at LLM architecture, training & fine-tuning, prompting, mechanistic interpretability of LLMs, LLM agents and different evaluation methods. Participants will be offered both a technical perspective and encouraged to critically think about important topics relevant to cognitive science and society in the context of LLMs.
Intended audience#
The course is intended for master students or advanced bachelor students (of, e.g., (computational) linguistics, cognitive science, or computer science) who are interested in language models / NLP / linguistics / AI / interdisciplinary approaches to state-of-the-art technical advances. Prior knowledge of LLMs is not required; prior experience with programming in Python is highly encouraged.
Course formalia#
In SS 2024, the course consists of a weekly lecture (Tue) and a weekly practical seminar (Fri) which will systematically introduce and cover concepts from the lecture hands-on, in practical worksheets, where participants will work with (small) LMs themselves. This webbbook will host both lecture materials and practical materials.
The course is intended for 6 ECTS. There will be compulsory homework assignments and a final exam. Optionally, for 9ECTS, students will conduct group projects.
Schedule#
The lecture overview below is preliminary and subject to changes.
Introduction
Neural networks and small language models
Transformers
Large language models, prompting & fine-tuning
Probing & attribution
Evaluation
Mechanistic interpretability
Further materials#
There are various courses on deep learning, NLP and LLMs out there. To name a few, these are excellent sources for further related materials:
Large Language Models by Ryan Cotterell’s lab
Large Language Models by Percy Liang’s lab
Pragmatic NLG with neural language models by Michael Franke
Andrej Karpathy has an excellent 1h overview video.
We also highly recommend Brian Christian’s book The Alignment Problem for a more general perspective on AI and computational CogSci.