(Roughly) Daily

A third and final (for awhile:) post on AI and what it might mean… Sunday’s post was a consideration of what we might lose if AI becomes central to our lives, a modern echo of Socrates’ lament on the spread of writing. Indeed, we are in a moment of high dudgeon over AI: its potential impact on the environment, on jobs. on the economy more broadly, and in the wake of recent statements from foundational model companies, on their governability(“rogue AI,” though do also see here), and thus on the prospect of our very existence (though do also see here).

Yesterday’s post was an example of thinking that starts from the implicit assumption that AI is coming, then asks how we might govern it. It’s worth noting that even if there is an AI Bubble about to burst, AI could still loom large in our longer-term future. Investors hobbled in many earlier bubbles (e.g., railroads, telecom, the dot.com crash, et al.) were directionally absolutely right about the longer-term promise of the technologies that excited them, just wrong to believe that the early entrants (and their specific offers) in which they were investing were the vehicles to realize that promise.

Today, we stick with the assumption that AI will continue to develop and to change the world around it. Some suggest that AI and its impact are moving us from literacy back to orality (the loss of which so concerned Socrates). But others suggest that AI (and the advances that it enables) could usher in a new Axial Age, a time of huge and ultimately positive transformation, on the other side of which we– humanity– will be… different, as different from the people we were in, say 1999, as Shakespeare was from Socrates.

But what might that be like? It’s relatively easy to imagine disasters that amount to the things we know collapsing; it’s much harder to envisage positive futures filled with altogether new things, futures the contents of which are, essentially by definition, things we’ve not imagined, much less experienced.

Following, from Vinkatesh Rao, one of the most thoughtful approaches I’ve yet come across. He starts with mathematics– with the recent successes of AIs in that field– locates that work in the history of math (in particular axiomatics), then suggests a way of understanding AI’s “success”:

… A theorem prover does not search the space of all possible sequences of logical symbols. It plays inside a world humans have painstakingly designed.

This makes recent achievements in AI mathematics look slightly different.

Perhaps we have been too quick to interpret them as evidence that AI has acquired access to the wild source of mathematical intuition. A significant part of the achievement may instead be that humans spent a century converting increasingly large regions of mathematical wilderness into exceptionally good game boards: semi-closed environments in which the relevant state can be represented, legitimate moves can be made, accumulated technique can be reused, and success can be verified…

Rao then explains how similar trajectories have accrued in the application of AI to programming and to physics, concluding…

Across mathematics, programming and physics, the same pattern keeps appearing. First there is clever performance. Then somebody finds a representation that captures what the clever performers are doing. Then general machinery accumulates around the representation. Then the machinery becomes architecture. Eventually much of the play becomes mechanical.

The creativity has not disappeared. It has moved.

There is an obvious objection to this story.

Perhaps what I am describing as progress is merely legibility.

James C. Scott famously described [and here] the tendency of modern states to simplify complicated social and natural realities into representations that administrators can see and manipulate: cadastral maps, standardized names, scientific forests, censuses, planned cities.

The representation is useful precisely because it throws things away. A forest seen by a forester interested in timber yield is not the forest experienced by the people, animals, fungi and plants living in it. The administrative forest has become extraordinarily legible along a narrow set of dimensions by becoming blind along others.

Every game performs a similar operation. Before you can play, you must decide what the pieces are. You must decide what constitutes the state of the game, what counts as a move, what consequences follow from moves, and what counts as winning. Everything else becomes environment.

A rigid body in a mechanical simulation is not actually rigid. A point mass is not a point. A frictionless surface does not exist. A program described by its types still runs on processors with caches and memory hierarchies. A theorem encoded in Lean does not contain the diagram that gave somebody the idea for it.

Formalization is compression.

In physics this is explicit. Reality becomes a model, the model becomes mathematics, and the mathematics becomes a numerical representation. Information is discarded at every step.

The miracle is not that nothing is lost. The miracle is how often we can throw almost everything away and still retain what matters…

…

… This suggests an important asymmetry. A productive mathematical ecology wants porous epistemic boundaries and impermeable verification boundaries. On the way in, metaphor, hallucination and category mistakes can be useful. On the way out, they are forbidden.

A poem can inspire the theorem. The poem cannot prove it…

… A domain becomes playable when enough of its phenomenological complexity can be compressed into something like a state; when there is some reasonably stable repertoire of actions; when interactions can be repeated; when outcomes provide feedback about better and worse play; and when the environment is stationary enough that lessons learned yesterday remain useful tomorrow.

None of these conditions needs to be perfect. Poker contains hidden information. Markets change their own rules. Hunting happens in an uncontrolled environment. Conversation does not have an explicit score.

Playability is a matter of degree.

Nor is playability the same as formalizability.

Children learn social games without writing axioms. Hunters learn landscapes. Merchants learn markets. Political operators learn institutions. Formalization is merely one unusually powerful technology for increasing playability.

So are money, measurement, domestication, standardization, simulation, bureaucracy and digitization. They do related things. They compress states. They stabilize interactions. They make outcomes comparable. They create memory. They allow experience to accumulate.

This suggests a hypothesis:

Far more of reality than we might reasonably have expected can be carved into semi-closed environments in which experience accumulates, performance improves through repeated play, and success eventually becomes sufficiently legible to automate.

Call this the Curiously Playable Universe hypothesis…

…

… If playability were merely a property of AI-friendly digital environments, none of this would be particularly interesting.

But the pattern is ancient. Consider agriculture.

Hunting and gathering takes place in a relatively wild learning environment. Animals move. Weather changes. Useful plants appear where they appear. The training distribution is supplied by nature.

Agriculture does something profound to the learning problem: It changes the environment.

Fields stabilize locations. Planting stabilizes cycles. Domestication alters organisms. Irrigation alters water availability. Storage alters time. Property regimes and markets stabilize incentives.

Humans did not merely become better at learning nature. They made nature easier to learn.

The wild game becomes domesticated.

Eventually industrial agriculture pushes large portions of the process toward automaticity: standardized breeds, standardized feed, controlled environments, mechanized planting and harvesting, precisely measured yields.

Then the game moves upward.

Instead of merely optimizing how an organism is cultivated, we begin optimizing the organism itself. Selective breeding becomes genetics becomes genomic selection and genetic engineering.

The old player becomes a game piece.

It is tempting to map this progression onto contemporary machine-learning terminology. Hunting looks vaguely like reinforcement learning in a difficult environment. Agriculture introduces something like shaped rewards and a controlled training distribution. Factory farming begins to resemble a regime of relentless verification against measurable outputs.

The analogy should not be pushed too literally. Human cultures have always involved teaching, imitation, norms and complicated reward systems.

But structurally the direction is unmistakable: increasing control over the state representation, action space, feedback signal and training distribution.

Commerce underwent another version of the transformation.

Exchange begins embedded in relationships, obligations, reputation, kinship and local knowledge. Credit can be intensely personal. Value is contextual.

Money performs an astonishing act of compression.

Heterogeneous goods and obligations become comparable through a common medium. Markets then stabilize arenas in which repeated exchange produces prices. Accounting makes states more legible. Contracts formalize future obligations.

Eventually finance builds games on top of the game.

A derivative can be a claim on the future value of another asset. Options put prices on possible future prices. Markets become arenas for expectations about expectations.

Play moves upward.

None of this required artificial intelligence. Humans have been turning wildernesses into games for thousands of years.

What AI changes is the cost of achieving playability in a domain.

Historically, making a domain playable was expensive. Somebody had to invent the categories. Somebody had to measure the variables. Somebody had to standardize the procedures. Somebody had to construct the institutions that made interactions repeatable.

And the resulting representation had to be relatively explicit because ordinary software was brittle.

Machine learning relaxes that requirement…

Rao muses on where this effect might accrue…

Coding is perhaps the clearest domain currently passing through the transition.

Programming was unusually playable before generative AI arrived.

Compilers already supplied verifiers. Type systems constrained legal moves. Unit tests supplied rewards. Version control recorded trajectories. Continuous integration repeatedly evaluated outcomes. Package ecosystems created enormous libraries of reusable moves.

Generative coding systems arrived in a landscape generations of programmers had inadvertently prepared for them.

As implementation becomes cheaper, the game moves upward toward specifications, architectures, product decisions and the increasingly important question of what software should exist in the first place.

Robotics is a more difficult frontier.

The physical world is phenomenologically unruly. Objects deform. Friction varies. Things break. Lighting changes. People walk into rooms. Drawers stick.

Simulation, cheap sensors, multimodal models and increasingly capable world models are gradually making physical environments more playable, but reality continually leaks through the representation.

This is why robotics may be one of the most important tests of the Curiously Playable Universe hypothesis.

Can enough of ordinary physical reality be compressed into stable learned representations to permit the same cycle of domestication and automation?

Science will probably fragment according to playability rather than according to our traditional rankings of intellectual difficulty.

Protein structure turned out to be unusually playable.

Drug discovery is less so because chemical promise must survive biology, organisms, clinical trials and human heterogeneity. Materials discovery may become increasingly playable through the combination of simulation and automated laboratories. Ecology may remain stubbornly wild because every useful abstraction excludes interactions operating at another scale.

Medicine contains both extremes. Image interpretation and molecular design can be made relatively game-like. Caring for an elderly person with five interacting conditions, family constraints and changing preferences is another matter.

Governance may become a particularly strange frontier.

States have spent centuries making populations legible through names, addresses, laws, property records, taxes, censuses, bureaucratic categories and standardized procedures. In that sense the modern administrative state is already a vast game-making machine.

AI will make more of its internal operations automatic.

But political systems contain an unusual source of wilderness: the pieces know they are pieces.

People respond strategically to measurements, categories and incentives. A rule changes the behavior it was intended to regulate. A metric becomes a target and stops being a good metric (Goodhart’s Law). Political actors learn to play the machinery designed to make them legible.

This may push governance upward toward new games involving states themselves: transnational protocols, financial systems, supply chains, standards regimes, platform governance and other structures that do not fit comfortably inside the old nation-state game board.

Some domains may resist for much longer.

Child-rearing is difficult to make playable because the objective changes as the child changes. Friendship has no stable score. Diplomacy involves adversaries whose models include models of your model of them. Entrepreneurship often consists precisely of discovering a game nobody realized existed.

Frontier science operates where we do not yet know which measurements matter. Cultural creation is evaluated by audiences who change partly because of the works being evaluated. Political legitimacy is altered by attempts to measure and optimize it.

These are not necessarily domains AI cannot enter. They are domains in which the wilderness fights back…

He continues…

… We keep asking what it means that machines can now play chess, write programs, fold proteins, prove theorems, steer robots and discover molecules. The question presumes that the remarkable new object is the machine—that some mysterious substance called intelligence has finally become “general” enough to flow from one human province into another.

But perhaps the stranger discovery concerns the provinces: Wildernesses can be reliably turned into game boards. We domesticate the territory, become skilled at its game, build machinery that plays it better than we can, and move outward to construct another game around the first, domesticating a larger scope.

This happened to fields and livestock long before it happened to chess. It happened to trade before it happened to programming. It happened to mechanics before it happened to mathematical proof.

Artificial intelligence did not invent this strange property of reality. It is merely making it difficult not to notice.

The deepest surprise of the AI era may turn out not to be that intelligence was easier to manufacture than we thought. It may be that the universe is far more playable than we imagined…

And he concludes with a consideration of writing…

… Writing sits somewhere awkwardly in the middle of the playability spectrum. It is obviously more playable than friendship or political legitimacy. There are stable artifacts, accumulated techniques, recognizable genres, repeatable operations and abundant feedback. Sentences can be revised. Arguments can be tested for consistency. Stories can be checked for continuity. Editors can compare two versions and usually say something useful about why one works better.

But writing lacks the feature that makes Lean such an extraordinarily good game board: a verifier. There is no kernel that accepts Middlemarch and rejects a bad novel. The relevant state is incompletely represented, the available moves are effectively unlimited, and the reward function wanders around outside the text in readers, institutions, historical circumstances and cultures that change partly in response to what gets written. Recent attempts to extend verifiable-reward techniques to writing therefore have to manufacture approximate evaluators out of principles and pairwise judgments rather than simply checking an answer. Writing is playable, but imperfectly and unevenly so.

The unevenness is becoming easier to see because AI is revealing which kinds of writing were already more game-like than we realized. A corporate memo, SEO article, product description, technical explanation or conventional news report has relatively strong constraints: purpose, audience, format, facts, length, house style, perhaps measurable outcomes. Formulaic genre fiction has a looser but still recognizable game board of beats, tropes, pacing, character functions and reader expectations. Even fiction once assumed to depend heavily on irreducible human voice is proving surprisingly susceptible to systematic generation and variation. Writing is not one game but a family of wilderness activities at very different stages of domestication.

At the other end lies writing whose purpose is partly to alter the terms by which it will be judged. As Walter Benjamin observed, “all great works of literature establish a genre or dissolve one.”

A genuinely new literary form, a strange essay, a foundational work of philosophy, or a piece of criticism that gives its readers a concept they did not previously possess cannot simply optimize against an existing reward function. Its success may consist in creating a new one. This is writing at its most wilderness-like, and perhaps why arguments about AI writing become confused when “writing” is treated as a single capability. Producing competent prose, satisfying a genre contract, developing an argument and inventing a form are different games with different degrees of playability…

… The meta-game emerging above writing may therefore involve constructing—and strategically revealing—the game boards on which execution takes place. If competent prose becomes cheap, more resources can be directed toward finding the question, assembling unlikely source domains, inventing the useful distinction, designing sequences of inquiry, recognizing when an analogy is productive, deciding what belongs together, and establishing the criteria by which a finished object ought to succeed. The writer becomes somewhat less like a person manufacturing sentences and somewhat more like an architect and player of generative constraints.

That does not mean prose becomes irrelevant. A game board badly realized is still a bad essay, just as a brilliant architectural plan does not eliminate the need for a building. Nor does it mean the higher-level activity is permanently reserved for humans. AI systems may themselves become increasingly capable of inventing representations, designing probes, discovering genres and constructing new games. The point is only that automaticity at one level does not end the activity. It displaces its frontier.

Writing may therefore be undergoing the same transition described throughout this essay, only messier and in public. Some of its old games are becoming startlingly playable. Some are approaching automaticity. New games are forming above them. And beyond those remains the poorly mapped territory from which the next game board might emerge…

Eminently worth reading in full: “The Curiously Playable Universe,” from @vgr.bsky.social.

And further: “Our Eukaryotic Moment,” in which Rao urges us to keep our minds open– to resist “premature ontological closure in conceptualizing AI.” A taste:

… About two billion years ago, life underwent a change in architecture. Until then, the planet was ruled by relatively simple prokaryotic cells: bacteria and archaea, tiny packets of chemistry bounded by membranes, carrying their genetic material directly in the same cellular space where much of the business of life took place.

Then something happened, or rather, several things happened in an order that remains uncertain. At some point, an ancestral archaeon cell entered into a permanent symbiosis with a bacterium capable of unusually powerful energy metabolism.

That bacterium eventually became the mitochondrion.

Somewhere in the same long evolutionary transition, genetic material became enclosed within a nucleus, separating the storage and regulation of hereditary information from much of the cell’s everyday chemistry. Internal membranes proliferated, cytoskeletons became more elaborate, and a new kind of cell emerged: the eukaryote….

… Eukaryotic cells became larger, more internally differentiated and capable of forms of organization unavailable to their ancestors. Much later, and independently in several lineages, some evolved multicellularity. Cells themselves became components of larger entities, differentiating and cooperating in increasingly elaborate ways. From the eukaryotic architecture eventually came forests, mushrooms, octopuses, hummingbirds, whales and us.

Eukaryotes were more than just “better bacteria.” They represented a different organization of life, capable of sustaining far more complex structures..

We may be living through an analogous transition now. The emergence of AI, in the particular form that it has appeared (deep learning), is arguably the eukaryotic moment in human cultural evolution, understood in memetic terms, with humans playing the role of mitochondria, and AI the role of the nucleus…

Meantime: “After We Stop Grieving AI: Cyborgs, Centaurs and Cyberpunks” (via Patrick Tanguay) See also: “Diving Deeper Into Digital Transformation” from John Hagel (source of the image above).

As the estimable Tim O’Reilly suggests, we’d probably be wise to take a breath, to treat AI as a “normal technology”: “Technology doesn’t force us… it merely opens the door.”

* Rainer Maria Rilke, Sonnets to Orpheus, Part Two, XII

###

As we crane to conceive what’s beyond our cocoon, we might recall that it was on this date in 1947 that the Association for Computing Machinery (or ACM) was founded. In one sense, computer science dates back to the 17th century and the mechanical calculators of Wilhelm Schickard and Gottfried Leibniz; in another, to the Difference and Analytical Engines of Charles Babbage (and the “programming” of Ada Lovelace). But the ACM can claim to be at least a key stone in the foundation of the field we now knw as “Computer Science”– the first modern “learned society” devoted to the emerging field. (Computer Science began to be established as a distinct academic discipline in the 1950s and early 1960s.  The world’s first computer science degree program, the Cambridge Diploma in Computer Science, began at the University of Cambridge Computer Laboratory in 1953. The first computer science department in the United States was formed at Purdue University in 1962.)

source