About
Foundation models are a revolutionary class of AI models that provides impressive abilities to generate content (text, images, sound, videos, protein structures, and more), and to do so by interactive prompts in a seemingly creative manner. These foundation models are often autoregressive, self-supervised, multimodal, transformer-based models that are pre-trained on large volumes of data, typically collected from the web. They already form the basis of all state-of-the-art systems in computer vision and natural language processing across a wide range of tasks and have shown impressive few-shot learning abilities. The perceived intelligence and adaptability of models like ChatGPT, Stable Diffusion, Flamingo, and GPT4 impress, but their aptitude to produce inaccurate, misleading, or false information (and present it confidently and convincingly) makes them unsuitable for any task of importance and poses serious societal concerns.
In this course we consider fundamental algorithmic questions concerning the multimodal learning and reasoning abilities of foundation models, as well as the ethical, legal, and social aspects of their deployment in society. Topics include, but are not limited to, the theory of foundation models, how foundation models appear to give rise to emergent behavior that was not observed during training, how the hallucination problem (as the lack of factual basis is referred to) arises and whether it can be mitigated by learning to retrieve and reference information from trusted sources, how to ensure foundation model objectives are aligned with their operators’ intentions, and how a more decentralized, compositional, and incremental approach to training and serving foundation models might reduce their data and compute hunger.
The course is research focused, with lectures on state-of-the-art advancements, reading and presenting of recent papers, and as final course deliverable a research-team project where students may study their own foundation model topic that should result in a research paper that is submitted to an academic venue.
This course is taught in the MSc program in Artificial Intelligence of the University of Amsterdam by Professor Cees Snoek and Assistant Professor Yuki Asano. The teaching assistants are Aritra Bhowmik, Mohammad Mahdi Derakhshani, Michael Dorkenwald, Yingjun Du, Danilo de Goede, Leon Lang, Phillip Lippe, Ivona Najdenkoska, Kien Nguyen, Mohammadreza Salehi, Zehao Xiao
