(Roughly) Daily

The estimable Tim O’Reilly argues that, while we’re right to be concerned with AI as it’s currently being developed, we needn’t resort to a Butlerian Jihad…

In 2019, the computer scientist Richard Sutton wrote a short essay called “The Bitter Lesson.” Looking back over seventy years of AI research, he concluded that general methods that scale with raw computation beat methods built on human expertise, and by a large margin. The AI industry as a whole is bitter-lesson-pilled, to borrow the internet’s slang for a true believer. I am human-pilled.

I believe in the bitter lesson. Brute force wins out in the end, and I have wrestled with that fact throughout my career. But we humans still have to figure out our role in the AI economy, and I find it rather disturbing that the AI community isn’t spending more time asking how to make this future friendlier for humans. The labs think about how to keep AI from going rogue, and their chief economists study job losses from AI. They don’t study what mechanisms we could introduce to create new economic opportunities for humans.

The question matters to us at O’Reilly Media. We don’t think of ourselves as a publisher, a conference producer or an online learning platform so much as a company that transfers knowledge from the people at the cutting edge to the people who want to follow them. So we are very interested in what human expertise is worth, even in the face of AI.

Human expertise was used to train AI, which makes AI a fabulous cultural technology. Thousands of years ago, the only knowledge people had was what they could keep in their heads. Since then we have built better and better ways to share knowledge across time and space, and AI continues that. With each of these transitions, we have to ask again what the truly human is still worth. People often say judgment, or taste. That’s true, but I think it goes deeper:

AI is a sea of possibility. All of that knowledge sits in the model, and nothing sets it in motion except a human. (That is changing a little as we start giving AI agents long-running tasks, and we will have to come to grips with it.) If you and I used Claude or ChatGPT for the same research project, we would each bring different experience to it, tell it what we want and why, and give it context in different ways. I see that as the unique focusing power of human consciousness and attention.

Earlier this year I spent an hour or so talking this question over with Claude. When it offered to write up a first draft for me, I said no. I wanted an essay from its own point of view about what it took away from our conversation. The result, “Why AI Needs Us”, opens, “He came in through Powys,” and a few paragraphs later says, “The choice of entry point was already the argument.” That captured what I’m getting at. I had come into the conversation through a book I love, John Cowper Powys’s The Meaning of Culture, which distinguishes culture from education. Powys quotes one definition: “Culture is what is left over after you have forgotten all you have definitely set out to learn.” It is what you absorbed. Each of us absorbs a small and particular slice of everything. Large language models (LLMs) try to absorb everything, and so become overly general; we give them focus. Human knowledge provides them with a lens...

[Tim unpacks his thesis, explaining both what he means and how he came to believe it…]

… I think a lot about a concept from economics called mechanism design, which has also been described as reverse game theory. Game theory asks how players will act under a given set of rules. Mechanism design starts from the outcome you want and works backward to rules that will tend to produce it as people pursue their own interests. Al Roth, who shared a Nobel Prize for this kind of work, used it to design a kidney exchange that more patients and donors would join. Economists mostly apply the idea to auctions and similar markets, but I think every new technology requires new mechanisms, and the best ones are not invented by economists. We tend to fall into them without really understanding them.

Think about the economic mechanisms that made earlier technological revolutions work. The web gave us the ability to connect computers, but it took inventions like PageRank to help people find what they were looking for, pay-per-click advertising to connect businesses with customers, and online payments to make transactions easy. Smartphones gave us GPS, but somebody had to invent the matching marketplaces that made Uber and Lyft possible. YouTube’s Content ID turned copyright infringement from a problem of takedowns into an opportunity for creators and rights holders to get paid. These weren’t simply technological advances. They were new arrangements of incentives that made entirely new markets possible.

So what are the equivalents for AI? How will agents discover the best expertise and skills, establish their quality, and pay the people who created them? How will they find services without the platform steering them toward whoever pays the most? We don’t know the answers yet. And that’s precisely the point. We’re spending enormous amounts of money making AI more capable, but comparatively little effort figuring out the mechanisms that will turn those capabilities into opportunities for everyone else.

Take open source software. Most people think its mechanism was the free software licenses, but I was around before those licenses, and people were already sharing software. Ken Thompson and Dennis Ritchie created Unix, the predecessor of Linux, at Bell Labs under AT&T’s proprietary license. They designed it to run on many kinds of hardware, and without meaning to, they built what I came to call an architecture of participation…

[Tim explains Model Context Protocol (MCP) and the way that it can bring participation to AI…]

…

With MCP, which lets any model connect to anyone’s tools and data, the emerging web of AI agents began to take on the shape of the participatory economy of open source software and the web. People can bring their own models and infrastructure as an alternative to the big centralized players.

This is another side of grappling with the bitter lesson: what are the missing mechanisms of the human-AI economy? Some of them are standards, because standards and protocols are a big part of what makes an architecture of participation. I have a group working on portable memory, so that what your AI knows about you can move with you from one tool to another.

That story is just beginning. We are in an exploratory phase of mechanism design for AI, and it would be a mistake to settle on answers too early. Somebody is going to figure out the next mechanism, and we’ll all say, “Oh, that’s so damn obvious.”

Mechanism design starts by asking what outcome you want. So what game ought we to be playing in the AI market?

We’ve been told that the game is the quest for superintelligence: a winner-takes-all contest that will bring untold riches to the winners and maybe impoverish, or even kill, everyone else. We’ve seen mechanisms designed to support that game. Massive investments in data center capacity mean that only the well-capitalized can play. There is colonialist-style appropriation of existing intellectual property. Access to models is sold through expensive APIs, and closed models come with terms of service that forbid others from learning from their outputs. Advertising may be creeping into answers, and memory locks your context inside one provider’s walls. Is that the game we ought to be playing?

I think the game we should be playing is human flourishing, and I think the game of human flourishing is creating new affordances for people to create and share with each other. We’re a social species. History teaches us that we will find our way there, but we can get there more quickly or more slowly depending on the choices we make. Getting there takes mechanisms that create a circulatory economy of value exchange between humans and their agents, with humans in partnership with AI rather than replaced by it.

The big labs, whatever the motives of the people who run them, are all driven by the imperatives of our financial system. They have raised enormous sums that they somehow have to pay back, so building a hugely successful business has to be their overriding goal, and that shapes all their incentives. In some ways AI was enshittified from the beginning, because it was built on the backs of so much capital that has to be repaid. That’s why open source AI matters so much, and sovereign AI too, built and controlled within individual countries. The open models, which anyone can download and run, aren’t as capable as the best closed models yet, but they keep getting closer, and they are already good enough for a lot of productive work.

Open source, open protocols, and open memory give users the freedom to shape and control the technology they use. The game I want us to play is one in which all of us build the future together, so that we control this wonderful new wave of technology for ourselves. I’m starting to see the shape of an economy where AI belongs to people rather than to the big labs, where we have personal AI agents working for us and with us, and we build an economy around that. I think it’s going to happen not because of the big labs, but in spite of them…

AI for everyone- the Bitter Lesson meets Architectures of Participation: “Human-Pilled,” from @timoreilly.bsky.social.

Pair with this consideration (from Henry Farrell) of AI’s current “flavor of the month”: “Agentic AI is myth set in motion.”

* Frank Herbert, Dune

###

As we open AI, we might recall that it was on this date in 1952 that the National Security Agency– the NSA– was formed by President Harry Truman. Created from a unit that deciphered coded communications in World War II, it had predecessors (with names like “Cipher Bureau,” “Black Chamber,” and “MI-8”) that date back to World War I. Charged with global monitoring, collection, and processing of information and data for intelligence and counterintelligence purposes, specializing in a discipline known as signals intelligence (SIGINT), the NSA is also tasked with the protection of U.S. communications networks and information systems.

By the end of the Cold War, the NSA had became the largest of the U.S. intelligence organizations in terms of personnel and budget. While it appears that CIA may now have surpassed it, the NSA has some 32,000 employees.

The NSA’s actions have been a matter of political controversy on several occasions, including its role in providing intelligence during the Gulf of Tonkin incident, which contributed to the escalation of U.S. involvement in the Vietnam War. Declassified documents later revealed that the NSA misinterpreted or overstated signals intelligence, leading to reports of a second North Vietnamese attack that likely never occurred. The agency has also received scrutiny for spying on anti–Vietnam War leaders and the agency’s participation in economic espionage. In 2013, the NSA had many of its secret surveillance programs revealed to the public by Edward Snowden, a former NSA contractor. According to the leaked documents, the NSA intercepts and stores the communications of over a billion people worldwide, including United States citizens. The documents also revealed that the NSA tracks hundreds of millions of people’s movements using cell phone metadata. Internationally, research has pointed to the NSA’s ability to surveil the domestic Internet traffic of foreign countries through “boomerang routing.”