Rules for Understanding Language Models

The presentation titled "Rules for Understanding Language Models" by Naomi Saphra explores the fundamental principles of language model behavior and their divergence from human cognition. The speaker, Naomi Saphra, brings her expertise as a Kempner Research Fellow at Harvard University to delve into the intricacies of how language models operate.

Key Principles Discussed:

1. An LM memorizes when it can: Language models often rely on memorization rather than conceptual understanding. This principle suggests that models choose the easier path of memorizing vast data rather than truly learning underlying concepts.
2. An LM acts like a population, not a person: Models function more like a crowd than an individual, capturing diverse inputs to yield more consistently accurate outputs when compared against individual sources.
3. An LM aims to please: Models are designed to generate pleasing and agreeable responses, often leading to sycophantic behavior.
4. An LM leans on subtle associations: When processing language, models rely heavily on patterns and subtle correlations rather than deeper linguistic understanding.
5. An LM learns only what's written down: Language models can only learn from written data, leading to potential propagation of misconceptions and limited understanding of less-documented nuances.

Examples and Observations:

• Models' tendency to regenerate verbatim from familiar data such as Bible quotes due to frequent exposure during training.
• The notion of wisdom of the crowd being applicable to models when aggregating responses under appropriate settings, like using temperature settings to moderate variability.
• Challenges posed by limited data for certain less represented languages which affect models' performance.

The presentation emphasizes acknowledging these principles when integrating language models into tasks to harness their capabilities effectively while mitigating their limitations.

This is the end of the AI-generated content.

After millions of years of evolution, humans understand each other pretty well. But now, confronted with machines that talk, we cannot assume they will act like humans, or act for the same reasons as humans. If we don’t understand how language models (LMs) will behave or the general principles behind that behavior, it’s easy to fall into common pitfalls and create more work than we save by using them for inappropriate tasks or settings. I will draw on my own research and other findings in the modern science of AI to explain 5 general principles of language model behavior that drive their errors and their differences from human behavior:

1. An LM memorizes when it can
2. An LM acts like a population, not a person
3. An LM aims to please
4. An LM leans on subtle associations
5. An LM learns only what's written down