Recent hype around large language models (LLMs) often cites "emergence" as a defining feature: capabilities that seem to arise suddenly as models scale. In "Large Language Models and Emergence: A Complex Systems Perspective," David and John Krakauer, along with Melanie Mitchell, challenge the vagueness of these claims and argue that true emergence has stricter, more quantifiable criteria. Drawing from decades of work in complexity science, the authors propose a more disciplined way to assess what counts as emergent behavior in artificial systems.

Redefining Emergence: Beyond Surprise

In the LLM literature, emergence often refers to unexpected task success or sudden jumps in benchmark performance. But Krakauer et al. argue this use is too loose. True emergence, they contend, involves not just novel output, but a coarse-graining process within the system—where internal representations become simpler and more effective at modeling external patterns.

Their framework identifies five key conditions for identifying emergence:

1. Scaling: New structures arise from increased size or complexity.
2. Criticality: Discontinuous changes appear at specific thresholds.
3. Compression: The system learns more efficient internal models.
4. Novel Bases: It develops new abstract units or representations.
5. Generalization: It applies internal models to new, untrained tasks.

Knowledge-In vs. Knowledge-Out

The paper distinguishes two forms of emergence: "knowledge-out" (KO) and "knowledge-in" (KI). KO emergence describes systems like fluid dynamics, where macro-behavior arises from simple rules among identical parts. KI systems, like LLMs, adapt through complex learning from a rich environment. Because LLMs are engineered and trained extensively, any emergent behavior must be scrutinized to ensure it's not just programmed complexity masquerading as spontaneity.

Are LLMs Actually Emergent?

The authors assess three types of emergence claims in LLMs:

1. Discontinuous capability jumps: Benchmarks like multi-digit addition show sharp gains with increased parameters. But these spikes may result from instruction tuning or in-context learning, not true emergent structure.

2. Capabilities not explicitly trained for: Claims of legal reasoning or analogical thinking often overlook the sheer breadth of LLM training data.

3. Emergent internal models: Experiments like OthelloGPT suggest LLMs form internal representations of external systems, but further analysis suggests these may be heuristic bags rather than true compressed models.

Emergence vs. Intelligence

Perhaps the most provocative argument in the paper is the distinction between emergent capability and emergent intelligence. The former may describe a system that performs new tasks due to scale. But intelligence, the authors argue, is marked by abstraction, analogy, energy efficiency, and parsimony—the ability to do more with less. LLMs, trained on massive datasets with enormous computational costs, may be powerful but lack the elegant, low-dimensional generalizations that define human reasoning.

Why It Matters

This paper arrives at a crucial moment in AI discourse. As tech companies race to build bigger models, there's a temptation to equate scale with sophistication. But Krakauer et al. caution against confusing brute-force capability with intelligence. If we want to understand and shape the future of AI responsibly, we must sharpen our definitions and resist the allure of vague analogies.

Final Thoughts

"Large Language Models and Emergence" is not a skeptical hit piece, but a call for scientific rigor. It invites us to distinguish between functional surprise and structural transformation, between engineered scale and spontaneous organization. The LLMs of today may impress us, but whether they truly "emerge" in the complex systems sense—or approach anything like intelligence—remains an open, and vitally important, question.