Hauptinhalt

Topinformationen

Study Project: Exploring Physical Understanding in LLMs (Part I)

8.3202

Dozenten

Beschreibung

Overview:
This study project aims to delve into the fascinating debate around whether Large Language Models (LLMs) can develop an understanding of the physical world. We'll explore the potential of multimodal LLMs, like GPT-4, which incorporate both text and visual inputs, to enhance their grounding in language and physical concepts.

Structure:
The project will be structured into three main phases, with weekly meetings focusing on different aspects of learning and research:

Part 1 - Background Learning:
Introduction to LLMs and their capabilities in next-word prediction and multimodal learning. Students will receive learning packages covering the foundational theories and the evolution of LLMs.
Deliverables: Each student will prepare a brief presentation summarizing the key concepts and their implications for understanding physical reality.

Part 2 - Literature Review:
Focus on discussing seminal and recent papers that are pivotal to understanding the current landscape of LLMs’ cognitive abilities, especially in physical understanding. This phase includes identifying gaps in the current research.
Deliverables: In small groups, students will present summaries and critiques of selected papers, highlighting methodologies, findings, and areas needing further exploration.

Part 3 - Research Operationalization and Benchmark Development:
Small groups will operationalize research questions based on identified gaps from the literature review. This phase will involve hands-on work with the GRAP evaluation benchmark as detailed in the paper available at https://arxiv.org/abs/2311.09048
Deliverables: Each group will develop enhancements or new features for the GRAP benchmark, aiming to address specific challenges in evaluating the physical understanding of LLMs. The project concludes with groups presenting their proposed solutions and implementations.