LSTMs & Transformers

Slides from the third lecture introducing a more advanced RNN architecture - LSTMs - and transformers - the architecture behind modern LLMs - can be found here.

Additional materials

If you want to dig a bit deeper, here are (optional!) supplementary readings, a general textbook on deep learning and the paper that first introduced backpropagation for training neural networks:

• Hochreiter & Schmidhuber (1997) Long Short-Term Memory

• Vaswani et al. (2017) Attention is all you need

• Devlin et al. (2019) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Radford et al. (2019) Language Models are Unsupervised Multitask Learners (GPT-2)