Times have changed. As the New York Times reports (in a piece apposite to Tuesday’s post)…
When Google prepared to go public in 2004, Larry Page, a co-founder of the company, wrote a letter to shareholders describing the internet firm’s responsibility to the world.
“We believe a well-functioning society should have abundant, free and unbiased access to high-quality information,” Mr. Page said.
Google fulfilled that responsibility by acting as a gateway to the internet. It answered people’s search queries with lists of hyperlinks, pushing users out to what is known as the “open web” — the millions of websites run by merchants, publishers, universities and others — for more information. In the ensuing decades, Google became one of the planet’s richest and most powerful companies by directing people to the vastness of the open web.
Now in the age of artificial intelligence, Google appears to be shrinking back from the open web — and may be imperiling it.
Since last year, the Silicon Valley giant has revamped its search with A.I. It introduced AI Mode, which replaces search results of hyperlinks with conversational responses written by Gemini, its A.I. chatbot. Most recently, Google changed its iconic search box for the first time in 25 years so that people could add photos and videos to their queries and assign A.I. “agents” to run searches for them.
The effect of these moves is becoming clear: People are spending more time with Google than ever…
… and spending less time with the sites that supply the information Google’s AI returns. Stephen Follows unpacks one web site’s experience– a cautionary tale for the open web and a reminder that, if we kill the sites generating the answers on which we depend, they won’t be there for anyone (directly or via AI) to find…
If you work in or around the film industry, there is a decent chance you have used the work of The Numbers this month, whether you realise it or not.
Its hand-researched data is the highest quality, tracking box office grosses, budgets, home video and streaming across more than 78,000 films and 236,000 people. It gets north of eight million visitors a year, and is treated as THE definitive authority by journalists, academics, filmmakers, prediction markets, and even Guinness World Records.
And it was this GOAT status which caused the catastrophic events of March this year.
The site was down for over a week, without explanation. A week later, it resurfaced at a fraction of its former size. Gone were the historical charts, the individual movie pages, and even the much-loved Report Builder.
With only a generic “we’re rebuilding, please bear with us” message to go on, the internet responded as it always does – with confusion, anger, and conspiracy theories. One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products.
Three months on, I spoke at length with Bruce Nash, founder and CEO of The Numbers, about what happened. He describes quite an unpleasant and eventful experience:
We got a lot of angry emails from people who are like, ‘Where’s this page that you used to have and you don’t have anymore?’
Within his tale are a number of things that should worry anyone who runs, relies on, or simply appreciates the internet…
[Follows recounts the history of The Numbers: its launch on Geocities, its growth, the onslaught of robots, the attack of hackers, and the coming of the LLMs and their relentless scraping…
… How bad could [AI bot scraping] be? Pretty bad, tbh. Enough that site owners such as Bruce have to question the value of something that will take so much time and money to build and defend.
Cloudflare, which protects a huge share of the world’s websites, publishes data on how many pages each AI platform crawls for every one visitor it sends back to the websites it crawled.
Google crawls about five pages for every visitor it sends you. OpenAI crawls over 1,000. Anthropic crawls over 38,000 pages for every single visitor it refers.
Note that the scale is logarithmic, i.e. each step along the bottom is ten times bigger than the last, because otherwise the differences are quite literally too large for me to include on one chart.
For the history of the internet to date, the principle of the open web was that, in return for letting the search engine robots read your site, they would send you readers. But now, that trade no longer applies. The number of robots has exploded, and they no longer send anyone back.
When this firehose is aimed at a small site, it can inflate the bandwidth bill and possibly even take down an entire site…
[Follows looks at other sites, including Wikipedia, in similar straights, and at attempts to protect these sites…]
… Whether any of this works depends on whether the AI companies play along rather than route around it. But as Bruce put it to me, somebody has to try.
Let’s look beyond the specifics for a moment and consider what happened here.
A beloved, useful, free website, run carefully by a competent, honest person for nearly thirty years, was crushed between two features of the new AI economy.
Unsustainable machine traffic hammered it from above, and in all likelihood a financially motivated intruder, operating in a world where breaking into websites has never been easier, took it down.
Bruce’s business and livelihood survived only because the website was not the whole business.
Others have not been so fortunate. Just last week, ZEGO, a German textile firm that had been in business for 37 years, filed for insolvency after a single cyberattack in March shut down its production for six weeks. Unlike The Numbers, they had no other business to fall back on.
The web is full of independent archives, hobby databases, local news sites, forums, reference works. Decades of accumulated human effort, running on old code, maintained by small teams or single individuals, quietly holding up far more of our shared knowledge than anyone acknowledges.
The Numbers is coming back, better built than before. I would encourage you to keep using it, keep supporting it, and, if you are one of the many people who emailed Bruce in fury about a missing page, perhaps send a kinder one now you know why it was missing…
As we conserve our culture, we might spare a thought for Vladimir Zworykin; he died on this date in 1982. An engineer and inventor, he is considered “the father of television” (or at least one of them). At the dawn of radio broadcasting, Zworykin began developing a system for transmitting sound and pictures. Other inventors were using a motorized, mechanical scanning system with rotating disks capable of a picture about one inch square– bulky, heavy, and impractical for home use. Zworykin, at Westinghouse, instead developed an electronic scanning television system using cathode-ray tubes; his innovations– among them, the iconoscope (the forerunner of the television camera) and the kinescope (which allowed the transfer of video to film)– laid the technical foundation for television as we came to know it… and contributed to the development of the electron microscope.
Your correspondent is a co-founder and director of Common Sense Media, an organization devoted to understanding the fraught media environment in which we live, to helping children, their families, and educators navigate it, and to advocating for “common sense” protections for kids. When we began, over 20 years ago, the terrain was largely TV, movies, books, and video games. They remain, of course, but the spotlight moved to social media, and now, to AI.
Advocacy, as it turns out, can be a tricky needle to thread. As Francis Bacon (and following him, a stream of others) observe, sometimes “the cure can be worse that the disease.” And that’s especially true when the “disease” one is fighting is the province of a powerful, richly funded cabal who market their moves as cures, even though, through their lens, disease is the desired state.
By way of one (chilling) example, Joe Wilkins outlines one dimension of the current state-of-play…
The White House and Congress are working on a deal to clamp down on what US citizens are allowed to say online. According to new reporting by Axios, the Trump administration is negotiating with key senators in an effort to shoehorn a massive legislation package which would limit states’ abilities to regulate AI in exchange for placing broad federal limits on digital speech.
Plenty of ink has been spilled about the Trump administration’s push to revoke AI regulation from individual states. Though the White House and its allies frame this as a matter of “safety” and “national security,” the timing is telling, coming as progressive state governments move to restrict the building of AI data centers and hold tech companies liable for harms their AI systems cause.
What makes this deal particularly insidious is the trade-off at its core. Per Axios, congressional lawmakers lead by Republican Marsha Blackburn are essentially offering to surrender their ability to regulate AI in exchange for three federal censorship bills: the Kids Online Safety Act, the NO FAKES Act, and a federal age verification mandate.
While the language around these three measures suggests common-sense cyber regulation, activists say they really amount to a massive censorship regime that is fundamentally anti-democratic.
In a blistering statement, the first amendment group Foundation for Individual Rights and Expression (FIRE) warns that “taken together, these bills would fundamentally change the internet as we know it.” That’s especially striking because FIRE is funded by conservative private interests like billionaire Charles Koch, meaning its opposition highlights the degree to which Trump is butting heads against fellow members of the US ruling class.
The Kids Online Safety Act (KORA) for example, would force social media giants to restrict lawful speech based on Federal Trade Commission regulations. FIRE complains that this would give the federal government too much power to hold platforms like Meta accountable for their harms — which, many would argue, is long overdue. But given that just a few corporations own the vast majority of web infrastructure and social media, KORA would give the Trump-controlled FTC major power to bend the internet to its will.
To put this power into perspective, just regulating Meta’s Instagram would impact about 71 percent of US citizens who say they regularly use the app.
It’s a powerful weapon, in other words — and like any weapon, it matters a great deal who wields it. Should the White House and Congress push KORA through, it’s likely it would functionally end the possibility of surfing the internet anonymously, while supercharging Trump’s efforts to criminalize left wing opposition groups in the US.
Whether it passes will depend on some shrewd maneuvering on the Trump administration’s part to secure Congressional support. But it all underscores a frustrating reality of AI regulation in the US: Americans overwhelmingly support stricter regulation on AI — but with the current cabal running the Oval Office, there’s no guarantee that the cure won’t be worse than the disease…
Common Sense Media continues to fight– both overt malignancies and the faux remedies that promise to be as bad or worse. So, I would argue, should we all.
As we seek sanity, we might recall that it was on this date in 1788, with the ratification of the ninth state (New Hampshire), that the Constitution of the United States came into effect. It was, of course, briskly amended to include what we call the Bill of Rights— the first 10 of 27 amendments (so far).
From yesterday’s post on the possible (and promising, but also potentially painful) future of computing to a pressing predicament we face today. The estimable Anil Dash on the threats to the open web…
You must imagine Sam Altman holding a knife to Tim Berners-Lee’s throat.
It’s not a pleasant image. Sir Tim is, rightly, revered as the genial father of the World Wide Web. But, all the signs are pointing to the fact that we might be in endgame for “open” as we’ve known it on the Internet over the last few decades.
The open web is something extraordinary: anybody can use whatever tools they have, to create content following publicly documented specifications, published using completely free and open platforms, and then share that work with anyone, anywhere in the world, without asking for permission from anyone. Think about how radical that is.
Now, from content to code, communities to culture, we can see example after example of that open web under attack. Every single aspect of the radical architecture I just described is threatened, by those who have profited most from that exact system.
Today, the good people who act as thoughtful stewards of the web infrastructure are still showing the same generosity of spirit that has created opportunity for billions of people and connected society in ways too vast to count while —not incidentally— also creating trillions of dollars of value and countless jobs around the world. But the increasingly-extremist tycoons of Big Tech have decided that that’s not good enough.
Now, the hectobillionaires have begun their final assault on the last, best parts of what’s still open, and likely won’t rest until they’ve either brought all of the independent and noncommercial parts of the Internet under their control, or destroyed them. Whether or not they succeed is going to be decided by decisions that we all make as a community in the coming months. Even though there have always been threats to openness on the web, the stakes have never been higher than they are this time.
Right now, too many of the players in the open ecosystem are still carrying on with business as usual, even though those tactics have been failing to stop big tech for years. I don’t say this lightly: it looks to me like 2026 is the year that decides whether the open web as we know it will survive at all, and we have to fight like the threat is existential. Because it is…
[Dash details the treats– largely, but not entirely driven by AI and its purveyors. He concludes…]
… The threat to the open web is far more profound than just some platforms that are under siege. The most egregious harm is the way that the generosity and grace of the people who keep the web open is being abused and exploited. Those people who maintain open source software? They’re hardly getting rich — that’s thankless, costly work, which they often choose instead of cashing in at some startup. Similarly, volunteering for Wikipedia is hardly profitable. Defining super-technical open standards takes time and patience, sometimes over a period of years, and there’s no fortune or fame in it.
Creators who fight hard to stay independent are often choosing to make less money, to go without winning awards or the other trappings of big media, just in order to maintain control and authority over their content, and because they think it’s the right way to connect with an audience. Publishers who’ve survived through year after year of attacks from tech platforms get rewarded by… getting to do it again the next year. Tim Berners-Lee is no billionaire, but none of those guys with the hundreds of billions of dollars would have all of their riches without him. And the thanks he gets from them is that they’re trying to kill the beautiful gift that he gave to the world, and replace it with a tedious, extortive slop mall.
So, we’re in endgame now. They see their chance to run the playbook again, and do to Wikipedians what Uber did to cab drivers, to get users addicted to closed apps like they are to social media, to force podcasters to chase an algorithm like kids on TikTok. If everyone across the open internet can gather together, and see that we’re all in one fight together, and push back with the same ferocity with which we’re being attacked, then we do have a shot at stopping them.
At one time, it was considered impossibly unlikely that anybody would ever create open technologies that would ever succeed in being useful for people, let alone that they would become a daily part of enabling billions of people to connect and communicate and make their lives better. So I don’t think it’s any more unlikely that the same communities can summon that kind of spirit again, and beat back the wealthiest people in the world, to ensure that the next generation gets to have these same amazing resources to rely on for decades to come.
Alright, if it’s not hopeless, what are the concrete things we can do? The first thing is to directly support organizations in the fight. Either those that are at risk, or those that are protecting those at risk. You can give directly to support the Internet Archive, or volunteer to help them out. Wikipedia welcomes your donation or your community participation. The Electronic Frontier Foundation is fighting for better policy and to defend your rights on virtually all of these issues, and could use your support or provides a list of ways to volunteer or take action. The Mozilla Foundation can also use your donations and is driving change. (And full disclosure — I’m involved in pretty much all of these organizations in some capacity, ranging from volunteer to advisor to board member.) That’s because I’m trying to make sure my deeds match my words! These are the people whom I’ve seen, with my own eyes, stay the hand of those who would hold the knife to the necks of the open web’s defenders. [Further full disclosure: so is your correpondent, and so have I.]
Beyond just what these organizations do, though, we can remember how much the open web matters. I know from my time on the board of Stack Overflow that we got to see the rise of an incredibly generous community built around sharing information openly, under open licenses. There are very few platforms in history that helped more people have more economic mobility than the number of people who got good-paying jobs as coders as a result of the information on that site. And then we got to see the toll that extractive LLMs had when they took advantage of that community without any consideration for the impact it would have when they trained models on the generosity of that site’s members without reciprocating in kind.
The good of the web only exists because of the openness of the web. They can’t just keep on taking and taking without expecting people to finally draw a line and saying “enough”. And interestingly, opportunities might exist where the tycoons least expect it. I saw Mike Masnick’s recent piece where he argued that one of the things that might enable a resurgence of the open web might be… AI. It would seem counterintuitive to anyone who’s read everything I’ve shared here to imagine that anything good could come of these same technologies that have caused so much harm.
But ultimately what matters is power. It is precisely because technologies like LLMs have powers that the authoritarians have rushed to try to take them over and wield them as effectively as they can. I don’t think that platforms owned and operated by those bad actors can be the tools that disrupt their agenda. I do think it might be possible that the creative communities that built the web in the first place could use their same innovative spirit to build what could be, for lack of a better term, called “good AI“. It’s going to take better policy, which may be impossible in the short term at the federal level in the U.S., but can certainly happen at more local levels and in the rest of the world. Though I’m skeptical about putting too much of the burden on individual users, we can certainly change culture and educate people so that more people feel empowered and motivated to choose alternatives to the big tech and big AI platforms that got us into this situation. And we can encourage harm reduction approaches for the people and institutions that are already locked into using these tools, because as we’ve seen, even small individual actions can get institutions to change course.
Ultimately I think, if given the choice, people will pick home-cooked, locally-grown, heart-felt digital meals over factory-farmed fast food technology every time…
As we protect what’s precious, we might send carefully-calculated birthday greetings to a man whose work helped lay the foundation for both the promise and the peril unpacked in the article linked above above: J. Presper Eckert; he was born on this day in 1919. An electrical engineer, he co-designed (with John Mauchly) the first general purpose computer, the ENIAC (see here and here) for the U.S. Army’s Ballistic Research Laboratory. He and Mauchy went on to found the Eckert–Mauchly Computer Corporation, at which they designed and built the first commercial computer in the U.S., the UNIVAC.
Eckert (standing and gesturing) and Mauchy (at the console), demonstrating the UNIVAC to Walter Cronkite (source)
The “Make Everything OK” button is a website containing nothing but a single button. Press it, and after a moment of processing, you’re informed: “Everything is OK now. If everything is still not OK, try checking your settings of perception of objective reality.”…
As we press and press again, we might recall that on this date in 1949, after two days in which a few flakes fell, Los Angeles “enjoyed” a real snow fall (the first that anyone can recall).
Snow at Jet Propulsion Laboratory, La Cañada Flintridge, January 1949. Photo courtesy of NASA/JPL Archive. (As this post is being written, JPL– a leading center of study of the science of wildfires– has been evacuated due to the encroaching Eaton fire.)
In 1840, British architect George Wightwick published a world history of architecture in the Romantic mode, inviting readers to enter a vast garden where Buddhist iconography rubs shoulders with Greek temples and Egyptian pyramids gaze upon Gothic cathedrals. His intended audience? Idle women. Matthew Mullane revisits this visionary but ultimately unpopular text and explores the legacy of attempts to gatekeep the realms of imagination and fantasy pertaining to the built environment…
The “Prince Architect” welcomes you: “You will see, within this domain, an epitome of the Architectural world. Mine is, as it were, a palace of congress, wherein you will be successively addressed by humble (but, it is hoped, characteristic) representatives of the great families of Design in ancient and Mahomedan India, China, Egypt, Greece, ancient and modern Italy, Turkey, Moorish Spain, and Christian Europe”.
This grandiose introduction is offered by the protagonist of George Wightwick’s Palace of Architecture: A Romance of Art and History (1840). The reader, an imagined visitor referred to in the second person, is quickly handed a map showing the “architectural world” not as a diagram of transmission, a “tree” of influence, or a catalogue of entries, but a picturesque garden. Flanking the central palace is a group of buildings representing the “ancient” corners of the world, including India, China, Burma, and Egypt. At the top-right corner of the map, Greek and Roman structures curl leftward to show a European panoply of styles including Gothic, Soanean, Greco-Roman, and finally, two pointed styles from the Christian and “Mahomedan” perspective. Before entering this garden, you face the palace gate, an unruly collage of world architecture history consisting of, among other things, a Gothic spire, an Islamic dome, and crude prehistoric stone. The gate represents the chasm between the Prince Architect’s overflowing storehouse of experience, and you, the new guest, with none. The well-traveled architect sourced the building’s components from his extensive travels and “crammed [it] with observation, the which it vents in mangled forms.”2 You, the reader, are homebound and observationally deficient and therefore must feel beguiled. However, after a guided journey through the grounds of the palace, “you will return, competent to read the significant details of what, now, only vaguely addresses your understanding.”
Unlike more familiar world histories of the nineteenth century that enticed readers with pages full of illustrations, simplified categorizations, and appeals to scientific rationality, Wightwick’s tour of world architecture was a poetically narrated experience. His florid language and direct reference to the reader were intended to “address the eye and ear of the general public with the eloquence of picturesque illustration and impassioned comment”. He believed that “the error of architectural authors has been that of writing technical treatises for professional readers” and approached the public with a different proposition: “[architecture] requires no critical knowledge of its beauties to admit; neither are its mathematics necessary to a certain enjoyment of the associations that may be connected with a building.”5 In other words, plans, geometry, and other artifacts of specialized knowledge are impediments to actually knowing architecture and its history, and a general audience requires none of that. What they need is a basic level of historical knowledge introduced in an evocative manner so that “the joy of being competent to appreciate” can be unlocked in order to experience “the poetic enchantment of Architecture [that] transfixes the soul of the beholder, and leaves him spell-bound under the combined influence of the phantom past, and the palpable present”. Instead of relying on the empirical evidence of professionals alone, after just one tour through the palace of architecture, you will be in command of architectural knowledge as your “own poet”.
Wightwick’s preference for “speculation and belief” over technical demonstration was directed toward a very specific readership: idle women. The seemingly neutral “you” that drew readers into the palace grounds was in fact aimed at the “fair countrywomen of England”. At a time when female readership of both popular and specialized material was growing, the book is perhaps the first world architectural history written specifically with women in mind…
…
… Critical responses to Wightwick’s entreaty to female readers spanned from bemusement to venomous reproach. The Gentleman’s Magazine recognized that the book was not for the “scientific observer” of architecture but acknowledged that it could nonetheless “afford amusement to the ‘fair’ and fashionable admirers of the art”. W. H. Leeds, writing under the penname Candidus, was not so generous. He published an excoriating review that held the book up as an example of the withering effects of Romanticism on contemporary architectural discourse. He called Wightwick the “wickedest dog in existence” for audaciously dedicating his book not to any reputable institute, but “to a woman, or a no-man” and thinking “that romance has anything to do with art—at any rate, with architecture”. Leeds argued that Wightwick’s avoidance of technical description, scarce reference to plans, and indulgence in imagination threatened to turn architecture into an unserious field of curiosity, charm, and play — all words invariably tinged with the feminine. If Wightwick’s book gained the influence its author wanted, then surely “that which has hitherto been the task of a higher order of intellect is now to become the amusement of women—perhaps the plaything of children”. Careful to not appear ungentlemanly, Leeds clarified that he is not opposed to women enjoying or appreciating architecture in a passive way, but Wightwick’s encouragement of active speculation and creative rearrangement of architecture history was dangerous. “We object to it”, he reasoned, “not because we question the capacity or the sex, but because we see no occasion for increasing the number of designing women”. Where Wightwick saw idle women as eager consumers, Leeds was concerned that an overly enticing history would shock them out of their idleness and convince them that they, too, could make architecture.
Men like Leeds feared that “designing women” would disrupt two key aspects of English architectural culture, its homosociality and its claim to truth. The first was perhaps an annoyance, but the second could be disastrous. Leeds argued that women’s flimsy associations and predilection for exclamations like “how exceedingly pretty!” could trigger the collapse of all architectural knowledge made by men before them.23 Such anxiety represents the panic of a discipline whose propulsive drive to include more and more case studies and accommodate more and more readers also brought unwelcomed actors, like women, dangerously close to the inner circle of architectural expertise. The discrediting of Wightwick’s book shows the quick hardening of professional and epistemological borders to maintain credibility…
…
… Into the twentieth century, architectural history remained stubbornly male dominated and the gendering of architectural fantasy and imagination as feminine stymied any hope that such ideas could gain professional credence. Things quickly changed in the 1960s, when young architects championed phenomenology as a critique of modernism’s universalizing assumptions about user experience. Historian Jorge Otero-Pailos argues that this phenomenological revolution allowed for a generation of so-called postmodern architects to challenge architectural history’s longstanding positivist bent. The buildings of Charles Moore, Robert Venturi, and Denise Scott Brown playfully assemble specific and invented historical references, provoking the viewer in a manner that is not so dissimilar from Wightwick’s mutant palace gate.
While postmodernists experimented with architectural history, the written output of architectural historians such as Charles Jencks ironically remained somewhat conventional and tied to the explanatory textbook. Fantasy and imagination seemingly still carried an indelible stigma. However, a few recent books suggest a return of the repressed, so to speak. Françoise Fromonot’s The House of Doctor Koolhaas (2025) tells the history of a famous house by Rem Koolhaas through the genre conventions of a detective novel. Charlotte Van den Broeck’s Bold Ventures: Thirteen Tales of Architectural Tragedy (2022) blends researched histories of architectural failure and suicide with self-reflective passages that question the authoritativeness of words like “explanation” that are so often used in history texts. Past authors are being rediscovered as well. Lin Huiyin’s mid-twentieth-century poems reflecting on a changing China are at last being translated and reframed as examples of architectural history. These texts are refreshingly strange — just as strange as walking into the Palace of Architecture — and signal that the discipline is finally shedding some of its enduring prejudices about imagination and fantasy…
As we dwell on design, we might send connected birthday greetings to an architect of a different kind, Sir Tim Berners-Lee; he was born on this date in 1955. A computer scientist best known as the inventor of the World Wide Web, the browser, the HTML markup language, the URL system, and HTTP, he is a professorial research fellow at the University of Oxford, a professor emeritus at the MIT, and director of the World Wide Web Consortium, which oversees its continued development.
You must be logged in to post a comment.