LSTMs & Transformers

Slides from the third lecture, introducing attention and transformers, can be found here. Furthermore, there are some excellent educational materials and talks out there – references to some original papers and some suggested resources are below.

Transformers: Part 2

Slides from the second lecture on transformers can be found here. The slides on mathematical formalization of transformers are available here.

Additional materials

If you want to dig a bit deeper, here are (optional!) supplementary readings, a general textbook on deep learning and the paper that first introduced backpropagation for training neural networks:

• Vaswani et al. (2017) Attention is all you need

• Devlin et al. (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Radford et al. (2019) Language Models are Unsupervised Multitask Learners (GPT-2)

• Series of lectures and talks on Transformers from Stanford

• Video on building GPT from scratch by Andrej Karpathy

Supplementary materials for the second transformers session:

• A blog / website on hardware in deep learning and LLM

• Rahwan et al. (2019) Machine behvaior

• Paper on positional information in transformers