Fine-tuning and RLHF

Slides of the session diving deeper into fine-tuning of LMs and, in particular, reinforcement learning from human feedback (RLHF), can be found here.

Additional materials

If you want to dig a bit deeper, here are (optional!) supplementary readings on some of the topics covered in class:

Supervised fine-tuning:

• Ding et al. (2023) Parameter-efficient fine-tuning of large-scale pre-trained language models

• Howard et al. (2018) Universal Language Model Fine-tuning for Text Classification

  • background paper rather than PEFT-related

RLHF:

• Blogpost about learning from human preferences (in the context of robotics) from OpenAI

• Ouyang et al. (2022) Training language models to follow instructions with human feedback

• RLHF overview blogpost from HuggingFace

• ChatGPT blogpost

• Bai et al. (2022) Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback