Posts Tagged ‘Henry Farrell’
“Unless you strip bodies of their corporeal nature and impart corporeal nature to incorporeal beings, nothing you expect will take place”*…

In 1965, Hubert Dreyfus wrote the first widely-considered critique of artificial intelligence, Alchemy and Artificial Intelligence, a report for RAND. He updated and expanded the work for broad publication in 1972 as What Computers Can’t Do (which was further updated and revised for two subsequent reissues) and a “sequel,” Mind Over Machine (in 1986). For a precis of Dreyfus’ views on AI, see here.
His invocation of alchemy resonates still, for example in the estimable Henry Farrell‘s consideration of Adam Becker‘s (great) recent book More Everything Forever…
Adam Becker’s “More Everything Forever” begins by describing the ideas of Eliezer Yudkowsky, an AI guru who Sam Altman thinks deserves a Nobel Prize. Yudkowsky’s ambitions for humanity include “[p]erfect health, immortality,” and a future in which “[i]f you imagine something that’s worse than mansions with robotic servants for everyone, you are not being ambitious enough.” According to Yudkowsky and his peers, a “glorious transhumanist future” awaits us if we get AI right, although we face extinction if we get it wrong.
“AI” and “transhumanist” are new terms for rather older ambitions. As the seedy occultist Dr. Trelawney remarks in Anthony Powell’s 1962 novel, “The Kindly Ones,” “[t]o be forever rich, forever young, never to die … Such was in every age the dream of the alchemist.” Renaissance alchemists won the support of monarchs like Rudolf II, the Holy Roman Emperor who squandered his realm’s money on a futile quest to discover the Philosopher’s Stone. Now, as Becker explains, AGI, or artificial general intelligence, has become the means through which philosophers might transubstantiate our mundane reality into a realm in which the apparently impossible becomes possible: living forever, raising the dead, and remaking the universe in the shape of humanity.
These ideas would be a curiosity, if they weren’t reshaping the world, and policymakers’ understanding of national security…
[Farrell explains, elaborating on the beliefs in play and explaining how they played in the Biden Administration (which, he argues, largely internalized them) and now in Trump’s…]
… The Biden administration’s policies rested on risky bets. AI may not be the universal solvent for vexing and complex technological problems promised by Altman, Amodei, and their peers. Some (including most members of the Association for the Advancement of Artificial Intelligence, as well as myself and colleagues) believe that current approaches will not lead to anything like self-improving AGI anytime soon. As DeepSeek’s success in building AI models suggests, semiconductors may not provide the chokehold on AI that American policymakers believed.
In any event, the Trump administration has seemingly opted for a different approach, replacing a grand plan for a global AI order with a loose approach based on the ever-shifting convergence between notions of American greatness and the particular interests of well-connected companies. This currently involves leveraging control of technology to maintain American domination, cutting deals with Gulf states, while trying to cut China out. When Trump toured the Gulf states with an entourage of tech CEOs, he offered access to top-end semiconductors in exchange for promises of massive investment, graciously accepting a jet plane, and a large-scale money transfer routed, with fees, through his family stablecoin.
This emerging blend of imperium, magical technology, and self-dealing too is reminiscent of the Renaissance. The English wizard John Dee, who claimed to have turned mercury into gold in the presence of Rudolf II, is also credited with coining the term “British Empire.” As a member of the court of Queen Elizabeth I, he combined alchemical experiment and indirect conversation with angels with more mundane plans to extend British power across the Atlantic and enrich himself.
The alchemists’ quest for immortality and universal transformation is again entangled with the struggle among empires. The dreams of perfect health, the conquest of death, and mansions with robot servants that Becker describes inspire visions of a radically transformed far future and a near present in which America can use advanced technologies to bolster its power. Such visions stem from an ideology of salvation that has indeed fueled important technological advances but may be quite wrong about where those advances will bring us.
We are being ushered into a world where AI models offer new and useful means to summarize the information of the world and draw nonobvious connections across it, but can also weave new syncretic religious beliefs together from the myriad human sources that they summarize. In the early 20th century, Weber argued that rationalized Calvinism had led to the defeat of the old gods and the inexorable disenchantment of a world dominated by bureaucracy and organized capitalism. In the early 21st, just the opposite is happening as AI draws open the gate for the return of the fantastical. Yudkowsky’s eschatology is just one of the prophetic visions attendant on AI. The practical changes that AI will bring to bureaucracy and markets will go hand in hand with religious ecstasy and fervor, as some people come to believe that they can speak to God through the machine, while others use it to elaborate their own extended fantasies about politics. As Becker’s book shows, technology is not just an abstract force changing the world, but a mirror in which human beings see their dreams and ambitions made manifest. We’re starting to discover what that looks like at scale.
Eminently worth reading full: “What AI debates have to do with alchemy,” from @himself.bsky.social.
Then consider this, from Nicklas Lundblad, a senior exec at Deep Mind, Google, Stripe, and elsewhere, recently turned public policy Fellow. In a piece adapted from a lecture in a series of talks on the philosophy of artificial intelligence that he gave at Mid-Sweden University), he puts Dreyfus’s thought into context, tracing the history of arguments with/against it over the years, as both technology and thinking about technology advanced. He concludes…
… So, we may say, what is Dreyfus answer?… That he is obsessed with showing that machines are incapable of being human?
In fact, his conclusion, already in 1965 – and then repeated in the 1972 book – is much more interesting. Here is how Dreyfus concludes his critique:
If the alchemist had stopped poring over his retorts and pentagrams and had spent his time looking for the deeper structure of the problem, as primitive man took his eyes of the moon, came out of the trees, and discovered fire and the wheel, things would have been set moving in a more promising direction. After all, three hundred years after the alchemists we did get gold from lead (and we have landed on the moon), but only after we abandoned work on the alchemic level, and worked to understand the chemical level and the even deeper nuclear level instead.
Dreyfus’ answer is not to get stuck in the question of what computers can’t do – but to reject even that last piece of thick ontology and ask if we are making a fundamental conceptual mistake. He suggests that we find what is to intelligence as chemistry is to alchemy and look for another set of concepts, another language, to describe what we are doing.
We would, in Dreyfus view, be much better of trying to build alien intelligence or replicate animal intelligence or even jettisoning the idea of intelligence entirely in favour of a new concept – perhaps something like cognicity – that removes the Difference from the research program.
Dreyfus recommendations would imply that we should drop the benchmarks that distract us and instead focus on simpler, thinner ontologies around tasks and interactions. What we need, he would suggest, is a conceptual shift into that simpler ontology – at least if we want to make real progress.
The shift into modern machine learning seems to prove that when the thicker ontology was abandoned we did discovered things obscured by the idea of man as a symbolic animal.
What if we abandon both the idea of man and machine?…
Also worthy of a full review: “Reading Dreyfus,” from @nicklaslundblad.
* Zosimos of Panopolis, an alchemist (and Gnostic mystic)
###
As we think about transmutation, we might recall that it was on this date in 1954 (only 11 years before Dreyfus wrote Alchemy and Artificial Intelligence) that IBM introduced the “IBM 608,” the first calculating machine to use only solid-state transistors instead of vacuum tubes (thus needing only about 5% of the power to perform the same operations). The machine was housed in several large cabinets and could be purchased for $83,210 (or rented for $1,760 per month)… that’s just over $1,000,000 on an inflation-adjusted basis (or about $22,000 per month). It used more than 3,000 germanium transistors and could store 40 nine-digit numbers and perform 4,500 additions per second, among other operations.
“Something is happening here / But you don’t know what it is / Do you, Mister Jones?”*…
Further to the post of Septmeber 15: on trying to understand what might lie on the other side of a transformation driven by AI (and related technologies). Microsoft’s Deputy CTO, Sam Schillace, suggests that organizations are about to change completely…
… Here’s a nerdy TL;DR if you want it: I think AI will enable a switch from what is essentially circuit switching today, in all kinds of organizations, to something that will look and feel a lot more like packet switching. This will happen partially because it can now (AI is to this problem what fast processors were to distributed networking protocols), and because networks are almost always more powerful, flexible, and scalable. This idea is going to feel super uncomfortable and be rejected by business leaders, employers and academics in much the same way that network engineers rejected the idea of packet-switched networks…
… In 1965, making a long-distance phone call in America was a feat of engineering coordination. A call from New York to Los Angeles required a dedicated copper path, physically reserved, for the duration of the conversation, and AT&T had to employ armies of switching engineers to manage this. The Bell System was one of the great technical achievements of the twentieth century, and the people who built it were not fools. They worked ferociously for what they were optimizing for: guaranteed, high-quality, reliable connections between two points.
It seemed fine – expensive, but that’s what everyone was used to. If you are my age, you remember when long-distance was even a thing, and those calls cost more and were precious. I still get a tiny bit nervous talking to my parents 3000 miles away in Michigan, that was so deeply ingrained. Part of me still can’t believe that phone calls all cost the same.
Then a small group of researchers, largely funded by ARPA, mostly working in universities, proposed something that sounded to the Bell engineers like a joke. Instead of reserving a dedicated path, you would break messages into small pieces, label each piece with a destination, and release them into the network to find their own way. Different pieces of the same message might travel completely different routes. They might arrive out of order. There was no guarantee that any particular piece would arrive at all. A phone call, under this scheme, would be like mailing a letter by tearing it into a hundred pieces and dropping each one in a different mailbox, hoping they’d all show up. (Kind of like training a giant AI model on lots of small fragments of language (tokens) and hoping that somehow intelligence would emerge from putting them back together randomly – which largely worked).
The Bell engineers had a name for this general idea: chaos. They had a specific critique too, which was that you couldn’t make any guarantees about quality of service. Voice needs low, consistent latency: you can’t have some syllables arriving seconds after others. And of course, data needs reliability: a corrupted file is useless. The engineers were right about all of this. Packet switching, in its early forms, was genuinely worse at the things circuit switching was designed to do.
It also, within a few decades, made the Bell System irrelevant. The internet didn’t improve on the telephone network; it made the telephone network a minor application running on top of something much larger. Today your phone calls travel as packets (which is why they all cost the same – there’s no such thing as a “long circuit” now), just like everything else. The Bell engineers optimized for the wrong thing. I love that this also has a flavor of “worse is better”. The Bell engineers were optimizing for “correct, complete, but complex and rigid” and the internet is more “simple and recoverable beats correct and complex”.
What actually happened was a philosophical shift about where intelligence should live. Circuit switching is a centralized control philosophy. Someone, somewhere, reserves the path before any message travels it. The network takes responsibility for delivery. The endpoints are dumb; they just send and receive. All the complexity, all the guarantee-making, lives in the infrastructure itself. This works, and it scales up to a point, and then it hits a wall, because the central system has to know everything about every connection, and that knowledge becomes impossible to manage as the network grows. (Sound familiar? This is how we build companies these days. We’ll come back to that idea)
Packet switching inverts this entirely. The network is dumb and the endpoints are smart. A router doesn’t understand the content of the packets it handles – it reads a destination address and forwards the packet one hop closer. The intelligence about what the message means, whether it arrived correctly, what to do if it didn’t – all of that lives at the edges, in the devices doing the communicating. The protocol defines what the endpoints must do. The network just moves bits.
You have seen this argument before, in a different domain. Central planning versus markets is the same disagreement. The Soviet planning apparatus tried to do for an economy what circuit switching does for a telephone network: reserve resources, guarantee outcomes, manage the whole system from the center. It failed not because the planners were stupid but because the information required to run a complex system centrally exceeds what any center can hold (it also failed because many of them were corrupt, and centralized systems are vulnerable to this, since there’s no way to build in mechanisms to correct, outcompete or otherwise reverse that corruption).
Prices, in a market, are packets. They carry information about supply and demand from the edges without anyone at the center having to understand or coordinate it. Markets are chaotic, wasteful, frequently unfair, and they consistently outperform central planning at scale for the same reason the internet outperformed the Bell System. The network always wins, eventually, because the network is where the information actually lives.
Let’s think about companies now. Companies love to do re-orgs. Or at least, they have to do them. They’re a lot of work, just like it was a lot of work to organize and change the old Bell network, and for the same reasons. Companies have to deal with the pain of organizational overhead, because to a large extent, they try to do central planning – circuit switching.
A manager, in an org chart, is a leased line. From an organizational perspective, they’re like a circuit between strategy and execution, reserved whether or not signal is flowing. The org chart is the circuit map. It tells you the paths that exist, which means it also tells you the paths that don’t. If you need to get information or work from point A to point B and there’s no line between them on the chart, you have a problem – you need to go up to a common node and back down, or you need to get someone to lay a new line. When a company needs to change how it behaves, it rewires the circuit map, which we call a reorg.
Why do we do this? This is not because companies are stupid or because the people running them lack imagination (hopefully). It’s because circuit-switched organizations are the only kind that could be managed without the infrastructure that packet-switched organizations require (which isn’t unlike what happened with the internet – we needed cheap processors to be able to build routers that could handle packets fast enough. Before we had that, we had no choice but to set up dedicated circuits).
If you can’t do the things an organization needs – observing and judging work, finding capabilities, holding people accountable – without the overhead of a fixed org chart, well, you just have to have it.
The cost of this solution is enormous and mostly invisible because it’s so normalized…
[Schillace expands on this idea…]
… When packet switching was proposed, the experts who objected most loudly were the most qualified people in the room. They were “right” in an engineering sense, but wrong in the systems sense. And they did what most people do, asking a “why not” question (against the old design and constraints) instead of a “what if” one: what if the new processors could enable a new network design?
Most people looking at AI are making the same mistake, and I predict we will see a lot of this from the orthodox “business advisor” community. I hear this from engineers and leaders already, and it sounds just like the Bell folks: You can’t guarantee accountability in a decentralized system. You can’t audit a network that routes around hierarchy. Regulators will never accept this. Sophisticated investors will raise concerns about governance.
All of these objections will be correct in the short term and beside the point in the long term. They are right from some kind of local engineering perspective but wrong from an emergent systems perspective. The question is not whether packet-switched organizations perform better on the metrics that circuit-switched organizations were built for. They won’t, at least at first, just like packet-switched networks didn’t at first. The question is whether the new architecture unlocks things the old one couldn’t do at all. And I think it will – I think we will get levels of scale, creativity and responsiveness by using these new tools to push intelligence to the edge of all kinds of organizations – not just businesses but political, social, educational and creative ones…
…There is a useful heuristic here, which is that disruptive ideas tend to produce a room that’s roughly split in half. Usually something like half the people think the idea is obviously correct (the folks asking “what if”), and half think it’s obviously wrong (the “why not” crowd). I’ve written about this kind of bifurcation before – this is the sign of something that is genuinely disruptive, and the split is between people who are rejecting it because it feels threatening, and people who have internalized the new world view and are beginning to extrapolate it…
[Schillace considers earlier, unsuccessful attempts at “packet-switched organizations,” concluding that “AI is [the] missing piece – cheap enough intelligence to put “everywhere” and enable a bunch of these ideas for real. It’s the equivalent of those cheap processors [that enabled packet switching] finally showing up, except it’s a much more complete stack arriving all at once” and that the next few years will consist in discovering the new protocols that will allow “packet-switched organizations” to succeed…]
… The Bell engineers who resisted packet switching were not, in the end, wrong about what they said. Voice quality did suffer in early VoIP, packets did get dropped a lot, and for a while, quality was worse. But they were wrong about what would matter in ten years, and catastrophically wrong about what would be possible in thirty.
Email was not a better telegram, the web was not a better encyclopedia, and streaming was not a better cable network. Each of these looks, in retrospect, like an obvious evolution, but none of them were conceivable from inside the circuit-switched world (I remember the freak out about how video was going to break even the highly scalable new internet. It wasn’t even thinkable in the circuit switched world). They required the new infrastructure before anyone could imagine what to build on it.
The organizational equivalent is similarly hard to see in advance. It is likely that packet-switched, AI-native organizations will not just do existing organizational things more efficiently, but that they will do things that are not currently possible. There might be organizations that form around specific problems and dissolve when the problem is solved, rather than persisting as structures in search of purpose, or expertise that routes to where it is needed rather than being allocated by headcount. We might have large groups with a degree of strategic flexibility and sophistication that have never existed together – we might get political or creative movements in radically different shapes because of that. We are definitely going to see an evolution in the creation, management and use of institutional memory. Training will probably look very different, as will job descriptions.
And almost certainly: we will get things we cannot name yet. Nobody in 1975 could have described Google, or Facebook, or VOIP, or even Google Docs.
The phone company’s engineers built something extraordinary. The Bell System was a genuine engineering achievement, and the people who designed it were some of the best technical minds of their era. They lost anyway. Networks are more robust, once the infrastructure to support them exists, and the right protocols to create and maintain them are adopted.
My bet is that the organizations of the future, the ones that compete and thrive, will make use of AI to build networks the same way the internet made use of those cheap processors and new protocols to build huge scale and new capabilities.
This will feel controversial, and might even be worse at first, but networks always win. The age of centrally planned organizations is coming to an end…
Eminently worth reading in full. Chaos that scales: “The Network Always Beats the Castle,” via @timoreilly.bsky.social.
See also: Benedict Evans‘ “AI, tools and transformation“:
… With every new technology, we start by using it for the work we already have, and we just do that more and faster. But then, over time, you make entirely new things. We will use AI to automate broad classes of stuff inside existing workflows and existing companies (although, as I’ve outlined above, that will be enormously more trouble and work than just giving everybody a model). But with every previous platform shift, the stuff that actually mattered was the stuff that wasn’t even possible before and that no-one even imagined…
And by way of keeping all of this in perspective, see Henry Farrell‘s “Machine god metaphors eat your brain” and Max Read‘s “Between 5 and 14 thoughts about AI and the discourse cycle.”
(Image above: source)
* Bob Dylan, “Ballad of a Thin Man” (from Highway 61 Revisited, 1965)
###
As we try to peer over the horizon, we might send carefully-calculated birthday greetings to a man whose work has helped propel the advances we’re pondering: James Hardy Wilkinson; he was born on this date in 1919. A mathematician, he is best known for his pioneering work in numerical analysis and numerical linear algebra, especially in the context of high-speed digital computing. In 1970 he won the ACM Turing Award.

“For every complex problem, there’s a solution that is simple, neat, and wrong”*…
Last year, in explaining the Biden Administration’s emerging new economic policy, National Security Advisor Jake Sullivan talked of a “small yard, high fence” approach to its trade with China. The idea: to place strict restrictions on a small number of technologies with significant military potential while maintaining normal economic exchange in other areas.
The estimable Henry Farrell argues that this approach to technology and China is working poorly (though, he suggests, it will work much worse if Trump wins and takes office in January). Self-reinforcing political feedback loops and self-reinforcing expectations are leading to breakdown.
The fundamental problem of managing geopolitics through manipulating technological trajectories is not readily solvable given existing means, Farrell suggests. We live in a much more complex world than existing state institutions are capable of handling. Therefore, he argues, we need to remake the state…
… Making the right choices in a complex policy environment requires an approach that is a world away from the application of brute force at scale. Your maps of the environment are going to be all wrong when you go in, and brute force is likely to have unexpected consequences. It isn’t just that you are going to make mistakes (you are), but your map of the actual problem you are trying to solve is likely to be utterly out of whack. As you try to catch up with China on EV, you discover that you don’t understand the market right. As you try to impose controls on military use of semiconductors, you find out that you don’t have the information you need to really actually understand how the semiconductor market works.
The problem – as Jen Pahlka’s book Recoding America explains at length – is that addressing such complex problems does not fit well with the way that the U.S. government works. When you are trying to impose order a vast sprawling bureaucracy, which is its own mid-sized global economy, and when your people don’t trust government much, you rely on rigid contracting systems, which define the problem in advance down to its finest details, even if that definition is out of whack with reality. You don’t build connections between the bureaucracy and outside actors, unless they run through cumbersome and rigidly pre-defined channels because it takes months or years to get approval for such connections. And you certainly don’t try to remake policy in realtime as your understanding of the situation changes. Pahlka’s book is cunningly disguised as an account of US software outsourcing practices. If it mentions either ‘national security’ or ‘economic security’ once, I don’t remember it. But it is arguably (along with Dan Davies’ similarly motivated The Unaccountability Machine) the most important book on these topics of the last twenty years. [Your correspondent heartily agrees.]…
… what do you do – is this. You start to think… about how to build economic security institutions that are designed from the ground up to manage complexity. If you want to take ‘small yard, high fence’ seriously as a policy approach, you need to build the apparatus to discover what lies inside, what lies outside, and what the barriers ought be. That apparatus – and its prescriptions – need to change over time both to match a better understanding of the policy environment, and changes in the environment itself.
And we don’t have the apparatus to actually implement small yard, high fence properly. Nor do we have it for pretty well every other plausible economic security policy you might imagine, short of a brute force decoupling of the U.S. and Chinese economies. And if you did that, you would need enormous capacity to manage the horrifically complex aftermath, if that aftermath could even be managed at all.
Clearly, it is far easier to make these arguments in the general than the particular. Saying that you need reforms is straightforward, but figuring out what they ought to be, let alone how to implement them in current political circumstances, is an altogether more difficult challenge. But it is where the debate needs to be going – and there is a role for technology in it. We are in a situation that rhymes in weird ways with the situation discovered by Vannevar Bush after World War II – recognizing that the needs of government had changed, that vastly better information and feedback systems were required to meet those needs, and that even if we didn’t exactly know what those systems were, we needed to start figuring them out, and quickly. That world had its pathologies. This one does too. But to prevent them becoming worse, we need better ways to manage them, and to ensure that the solutions are better than the problems that they are supposed to mitigate.
This is – obviously – a radical set of claims. But it’s one that is entailed by the diagnosis of the problem that I’ve presented. If we need to manage complex challenges – of which the U.S. China relationship is only one – we need a state that is capable of managing complexities. We don’t have one. And that remains a first order problem, regardless of however hawkish or dovish you are…
We need a new kind of state for the new geopolitics: “‘Small Yard, High Fence’: these four words conceal a mess,” from @himself.bsky.social (and @pahlkadot.bsky.social). Eminently worth reading in full.
* H.L. Mencken
###
As we ruminate on restructuring, we might recall that it was on this date in 1954 (7 years after the transistor was developed at Bell Labs) that Texas Instruments introduced the Regency TR-1, the first commercially-manufactured transistor radio. Its performance was mediocre, but its small size and portability drove sales of over 150,000 units.
Further to Farrell’s and Pahlka’s points, it’s instructive to ponder what became of the transistor radio as a product category (and of the competitors in it) over the next few decades– and the altogether-unanticipated plethora of small, convenient, hand-held product categories it spawned: calculators, mobile phones, tablets… and whatever comes next…







You must be logged in to post a comment.