Welcome to this week’s Annotations! This week we came across wide-ranging papers: from decoding the interplay of visual cortical areas to pioneering advancements in antibody design, from LLMs to… well, more LLMs. In an exciting plot twist, one author claims that transformers are actually GNNs; we ask (but don’t answer) the hard-hitting question: does that mean that GNNs are actually transformers?We are left wondering where GTNs fall in this perspective. Another paper applies sparse autoencoders—famous for unlocking Golden Gate Claude—to a protein language model (ESM2). And finally, this week we found not one but two papers that challenge and update age-old adages.
Visual neuroscience has traditionally focused on understanding individual cortical areas in isolation, which overlooks the interconnected nature of the visual system and the shared or unique representational content across these areas. Gifford et al. developed “relational neural control” to explore representational relationships across visual cortical areas, revealing how shared and unique content varies based on cortical distance, categorical selectivity, and hierarchical position, and validated these findings with in vivo fMRI responses.
The old adage that any publicity is good publicity might extend to the world of peer review. A new paper shows how peer review isn’t just a mechanism for evaluating research but also for spreading ideas. Even when scientists decline to review a paper, seeing the title and abstract increases their likelihood of citing the author’s work by 14%. Think about that as you cry through your NeurIPS reviews this week.
AI has been anticipated to revolutionize drug discovery, yet AI-designed drugs have seldom progressed beyond preclinical stages. Marinka Zitnik reports on a study by Xu et al., who conducted a groundbreaking phase 2a clinical trial demonstrating the safety and promising efficacy of rentosertib, an AI-discovered drug for idiopathic pulmonary fibrosis, marking a pivotal moment in AI-enabled drug discovery by advancing from target identification to clinical testing in under 30 months.
Zhu et al. demonstrate that a two-layer transformer utilizing continuous chain-of-thoughts (CoTs) can efficiently solve the directed graph reachability problem by encoding multiple search frontiers in a superposition state, outperforming discrete CoTs that require significantly more steps due to their sequential search limitations.
In a fascinating exploration of neural architectures, Joshi argues that transformers, originally designed for NLP, can be understood as Graph Neural Networks (GNNs) operating on fully connected token graphs, where self-attention acts as a message-passing mechanism, highlighting the serendipitous alignment of Transformer efficiency with modern hardware capabilities. If transformers are GNNs, does that mean that GNNs are transformers? 🤔
Protein language models (pLMs) have become essential tools for predicting protein structure and function, but the specific features they learn remain unclear. Adams et al. trained sparse autoencoders (SAEs) on the residual stream of ESM-2 and discovered a combination of generic and family-specific features that represent proteins. This bio-MechInterp work offers new insights into protein biology and should aid in the generation of hypotheses for unknown biological mechanisms.
Lu et al. demonstrate that transformers initially trained on natural language can efficiently generalize to diverse non-language tasks, such as numerical computation, image classification, and protein fold prediction, by fine-tuning only the input and output layers, showcasing its potential as a universal computation engine.
Koster et al. explored a classic economics problem of how to allocate resources from a common pool in a way that encourages people to give back more than take. They used RL to train AI agents to mimic human behavior and created a virtual economy to test different strategies. The models found a policy to exclude selfish players but forgive them quickly. So the age-old adage (another adage!) may need to be amended to forgive and forget… quickly. When testing on humans, this mechanism outperformed standard economic strategies, suggesting AI can help design economic mechanisms to balance punishment for free-loading with strategic generosity when resources are ample.
This LangGraph tutorial teaches how to create an ambient email assistant from the ground up, guiding you through the essentials of agent development while leveraging LangSmith to assess its efficacy. While an email assistant might be truly useless, the framework should inspire more creative agentic tasks.