Evaluation & behavioral assessment

This session covers the important topic of trying to evaluating how well LLMs do with respect to various tasks and evaluation methods, with the goal of assessing the quality and the performance of the LLMs. Slides of the session can be found here.

Additional materials

If you want to dig a bit deeper, here are (optional!) supplementary readings. More papers discussed in the lecture are provided in the slides.

• SyntaxGym benchmark for testing syntactic generalization of LMs

• Giant overview paper of Holistic Evaluation of Lnaguage Models (HELM)