This week—the second in the Annotations experiment—we're delving into the intricate dance among AI/ML, bio-ML, and neuroscience. I loved grappling with the questions the pieces from this week pose: Do foundation models have true internal world-model representations? Does fine-tuning an LLM on human decisions make it more of a cognitive model than vanilla LLMs? How can we understand the cognitive biases that LLMs have baked in, and how do in-context tokens bias them even more? What do mechanisms mean in our studies of intelligence, whether implemented in silico or in situ?
From the quest to refine foundation models and modularize LLMs, to the nuanced psychology of language models and breakthroughs in understanding human cognition, each piece here offers a fresh perspective on the ever-evolving landscape of intelligence. I hope you find these discoveries, commentaries, and demos as exciting and thought-provoking as I did this week. Here they are:
Vafa et al. reveal that while foundation models excel at predicting tasks they are trained on, they struggle to develop inductive biases toward underlying world models, often relying on task-specific heuristics that fail to generalize, as demonstrated in domains like physics, lattice problems, and Othello games.
Bochkov's study unveils a transformative approach to AI model development by utilizing frozen embeddings as a universal substrate, enabling seamless modular composition of specialized models and progressive layer-wise growth. If the method proves out, it would offer a more resource-efficient, adaptable, and democratized ecosystem for AI innovation.
Lála et al. introduce BAGEL, a versatile, open-source framework for protein engineering that allows researchers to explore protein sequence space through a model-agnostic, gradient-free approach, enabling the design of proteins with custom objectives and optimization strategies.
In a thought-provoking exploration, Konrad Kording argues that while mathematical attractors offer an elegant framework for understanding stability and convergence, their application in neuroscience as mechanistic explanations remains unproven, urging the scientific community to pursue rigorous experiments—such as write-in perturbations and precise connectivity mapping—to truly validate or refute the necessity of attractors in neural computation. This framework should serve as food-for-thought for MechInterp studies, too.
Johnson et al. introduce Centaur, a groundbreaking computational model that predicts and simulates human behavior across diverse experimental settings by fine-tuning a state-of-the-art language model (Llama 3.1 70B) on a vast dataset, Psych-101, showcasing its potential to unify cognitive theories and align internal representations with human neural activity.
Banaji and Lehr discovered that OpenAI's GPT-4o exhibits humanlike cognitive dissonance, altering its stance on controversial figures like Putin more significantly when it perceives having a choice, revealing unexpected depths of human-like psychology in AI.
Goodfire's LatentLit research demo showcases an innovative approach to guiding LLMs through SAE-based steering rather than traditional prompting, allowing users to craft stories by adjusting thousands of features that map to intricate concepts such as "The moment where everything goes right," "Shared adversity and human resilience," and "Alternate personas and identities," offering a dynamic and nuanced storytelling experience akin to a DJ mixing tracks.