(Roughly) Daily

Phil Tinline on the history and consequences of the impulse to optimize businesses, markets, and governments…

Optimization means achieving a measurable objective—that is, maximizing or minimizing a given number. It requires controlling the inputs and processes that affect the objective, in order to find the most efficient way to achieve it. It draws on centuries of mathematical discovery, but it emerged in its modern form under the pressure of World War II and the need to manage the byzantine complexity of US military logistics.

In the summer of 1947, a Stanford-trained mathematical scientist named George Dantzig sat in his office at the Pentagon, laboring over US Air Force planning issues with a desk calculator. The problems he had dealt with during the war and since involved “an astronomical number of feasible solutions to choose from,” making it impossible to calculate which was the best. As he recalled, “Those in charge often do a hand-wave and say, ‘I’ve considered all the alternatives,’ but this is so much garbage.” All those leaders had to offer was that their “‘experience’ and ‘mature judgment’ would guide the way” by laying down rules that would limit the options.

The problem, Dantzig realized, was that “you could never find any direct relationship between the stated goal and the actions to achieve the goal.” The solution, he believed, was to formulate a complex real-world problem as a mathematical model. This could then be solved by what became Dantzig’s “simplex algorithm” (or “simplex method”), provided a precise goal was set as its “objective function.”

The simplex algorithm radically reduced the number of feasible solutions. It soon became clear that it could be brought to bear far beyond the military: “All one had to do,” Dantzig remembered, “was change the names of the columns and the rows, and it was applicable to an economic planning problem or to an industrial planning problem.”

In engineering, optimization was put to use in designing rockets and aircraft, and the shape of cars, wind turbines, and hydrofoils. It redefined manufacturing, circuit design, and the management of supply chains. Its impact is still visible in measures of the occurrence of the word “optimization” itself. Before 1950, the term was barely in use at all; thereafter, its frequency soared.

Economists embraced optimization too. An early application was in the development of “portfolio theory,” which, as the aerospace engineer Joaquim R. R. A. Martins and the computer scientist Andrew Ning put it, “formalized the idea of investment diversification, marking the birth of modern financial economics.”⁷ One important element of optimization in economics is the inclusion of constraining factors: Given a set level of income or cost, how can we maximize utility? But economics is not quite as scientifically determinable as engineering—it’s more exposed to messy, contradictory fellow humans. Here, optimization starts to look rather suboptimal…

… As computers have become ubiquitous, optimization has spread ever deeper into human life. In 2021, a trio of Stanford academics published a book titled System Error: Where Big Tech Went Wrong and How We Can Reboot. They observed: “What begins as a professional mind-set for the technologist easily becomes a more general orientation to life. … The paramount goal becomes removing friction from everyday activities, automating repetitive tasks, and finding ways to save time while improving outcomes.” US tech companies, for instance, are often led by software engineers, who manage their staff accordingly, measuring results against precisely set objectives. An over-dominant engineering mindset, System Error argued, is extending optimization beyond the areas where it can be effective.

This might surprise the tech analyst Dan Wang, whose 2025 bestseller Breakneck: China’s Quest to Engineer the Future argued that “an American elite, made up mostly of lawyers, excelling at obstruction” had much to learn from China’s “technocratic class, made up of mostly engineers, that excels at construction.” But even Wang admitted that engineering logic can be taken too far. “Sometimes, it feels like China’s leadership is made up entirely of hydraulic engineers,” he wrote, “who view the economy and society as liquid flows, as if all human activity—from mass production to reproduction—can be directed, restricted, increased, or blocked with the same ease as turning a series of valves.”

Given its mathematical foundations, optimization depends on having numerical data that can be adjusted to achieve the numerically expressed objective. As the American historian of science Theodore Porter showed in his 1995 study Trust in Numbers: The Pursuit of Objectivity in Science and Public Life, governments began to gather data at scale and to rely on it for decision-making for reasons similar to those set out by Dantzig—to get away from the subjective judgment of leaders.

However, Porter warned that while using numbers to exercise power objectively might be an attractive idea, it is also impossible to do. Even governments can’t count everything, and choosing what to leave out is an intensely political decision. Worse, Porter wrote, “numbers have often been an agency for acting on people, exercising power over them,” even turning people “into objects to be manipulated.” As Wang noted, in China the drive to meet numerical targets has sometimes taken a crushingly simple form, as with the government’s “one child” or “zero Covid” policies.

Even when optimizers aren’t sealing sick people in their homes, as the Chinese state did during the pandemic, they are often so focused on their objective that they don’t notice the damage they’re doing. Whatever is not relevant to the objective can be shrugged off as a so-called externality. Witness corporations optimizing their operations to maximize profits or the price of their shares. Some squeeze pay or working conditions; others pollute with abandon, or exploit their dominant market position to force down their suppliers’ prices, regardless of the impact.

And the problem with optimization is not just a matter of unfortunate side-effects. We are seeing the emergence of what we might call “social optimization”—the belief that this idea offers a way to transform society as a whole. But as Porter’s work suggests, this is not a matter of neutrally making things better. Optimization privileges the measurable over the unmeasurable. And it places the onus for improving society on the ever-striving individual rather than asking more fundamental, structural questions about why systems work as they do and whom they empower and disempower.

This is not an explicit ideology. No doubt, businesses and governments often are simply following the logic and opportunities implicit in new digital technology, from smartphones to the cameras and sensors that can now cheaply and efficiently monitor a wide range of activities. Nonetheless, as new technology has made it possible to gather ever more numerical data, optimization has begun to embed its implicit values into our lives.

In the workplace, this can swiftly make people’s lives worse. Particularly in sectors such as logistics, new technology allows employers to optimize more and more rigorously for maximum productivity and minimum cost. It has become commonplace to give employees an ongoing score, with the aim of incentivizing them to compete continually. This goes beyond even the monitoring of worker efficiency that the management consultant Frederick Taylor pioneered in the early twentieth century and the numerical key performance indicators that his successors promoted. The intensive quantification of employees’ performance has come to be known as “digital Taylorism.”

Optimization has refocused the media around the measurable preferences of the individual, as tallied in clicks, page views, unique browses, and similar metrics. This erodes the shared moments that build a culture and the shared truths that underpin democracy. Social media takes this even further: Algorithms are optimized to maximize attention, incentivizing people to respond to political issues not with thought but vivid expressions of feeling, rewarding users numerically in follows, likes, and shares. Meanwhile, tracking apps increasingly normalize the optimization of health metrics.

Yet technologists are keen to go much further. Off the back of their successes producing software, they are raising their sights to the horizon, optimizing for a few grand objectives at all costs, in pursuit of an ever more perfect world. They have formed an alliance with philosophers and philanthropists in the Effective Altruism movement, which aims to purge generosity of the influence of feeling in favor of calculable reason—even as it confidently prophesies the far future. Other tech leaders support the principles of the “network state.” According to the journalist Gil Duran, this concept proposes to create “private, corporate-controlled cities” that will liberate innovators from the constraints of the democratic state and its messy, unmeasurable trade-offs.¹³

And most of all, the dream of social optimization reverberates through promises of an AI-transformed future, in which once unthinkably efficient tech will supposedly liberate individual human potential. In “The Techno-Optimist Manifesto” (2023), the venture capitalist Marc Andreessen proposed using technology and the free market to maximize abundance to the point of infinity. Though Andreessen insists he does not believe in “the Unconstrained Vision of Utopia,” he dismisses the “Precautionary Principle” as an “enemy.”

The problem here is obvious to anyone not immersed in the culture of Silicon Valley. Not every worthwhile objective can be measured. How do we quantify social peace, for instance, or the health of our arts and culture, or the concentration of power? Or the worth of work itself, or a truly enriching education, or kindness? We might hope the realization that not everything can be measured would prompt the promoters of social optimization to accept its limitations and appreciate the qualities of more deeply rooted systems, such as democracy. Alas, they tend to conclude that if a goal or a problem has no measure, it is not worth bothering with. Kevin Kelly, a technology journalist and an apostle of the Quantified Self movement, has faced criticism, as he puts it, that “only intangibles like meaningful happiness count.” His response: “Meaningfulness is very hard to measure, which makes it very hard to optimize.” Similarly, in a critique of the US anti-monopoly movement, the journalist Matthew Yglesias has protested that “‘corporate power’ doesn’t mean anything” on the grounds that it “doesn’t add up to anything measurable or actionable.”

But this is not the first time similar-sounding criticisms have been raised against attempts to perfect society. Where they chose to focus their fire, and where they didn’t, reveals what’s distinctive about the phenomenon of social optimization…

[Tinline reaches back to the early 19th century (and the earliest known use of the word “optimize”) then follows the development of what has become a powerful mindset– in effect, a movement. He concludes…]

Governments today do have something to learn from Dantzig’s insistence on the importance of having a clear objective. But his overly dim view of leadership needs to be constrained. It is increasingly clear that, in the less calculable areas of life, a leader exercising human judgment is preferable to an implacable optimization algorithm. Without such human-centered constraints in place, social optimization won’t make things better—only more extreme.

On not letting the perfect be the enemy of the possible– and often the preferable: “The Cult of Optimization,” from @philtinline.bsky.social in The Ideas Letter.

We should note that the optimization craze has taken hold at the personal level as well, with similar results. See. e.g., “Optimising is just perfectionism in disguise. Here’s why that’s a problem” and “Optimization Culture Is Making Us Miserable.”

* Donald Knuth (the godfather of computer programming)

###

As we celebrate slack and internalize externalities, we might spare a thought for a man who looked beyond the metrics od his day, Clifford W. Beers; he died on this date in 1943. An author and psychiatric patient, he is best known as the founder of the American mental hygiene movement.

Clifford Whittingham Beers was an American author and social reformer who wrote an autobiography documenting appalling conditions and maltreatment by staff of mental patients. His classic book, A Mind That Found Itself (1908) raised public consciousness of the need for reform. He had already himself experienced treatment as a mental patient, first in 1900, diagnosed with depression and paranoia. His four siblings also suffered mental health problems and died in mental hospitals, as he also did. In 1909, Beers founded the National Committee for Mental Hygiene (since renamed as Mental Health America) with the mission to improve the treatment in mental health institutions. By 1913, he was able to establish the Clifford Beers Clinic in New Haven, an outpatient mental health clinic, the first of its kind in the U.S., which continues his legacy to the present. – source

source