Times have changed. As the New York Times reports (in a piece apposite to Tuesday’s post)…
When Google prepared to go public in 2004, Larry Page, a co-founder of the company, wrote a letter to shareholders describing the internet firm’s responsibility to the world.
“We believe a well-functioning society should have abundant, free and unbiased access to high-quality information,” Mr. Page said.
Google fulfilled that responsibility by acting as a gateway to the internet. It answered people’s search queries with lists of hyperlinks, pushing users out to what is known as the “open web” — the millions of websites run by merchants, publishers, universities and others — for more information. In the ensuing decades, Google became one of the planet’s richest and most powerful companies by directing people to the vastness of the open web.
Now in the age of artificial intelligence, Google appears to be shrinking back from the open web — and may be imperiling it.
Since last year, the Silicon Valley giant has revamped its search with A.I. It introduced AI Mode, which replaces search results of hyperlinks with conversational responses written by Gemini, its A.I. chatbot. Most recently, Google changed its iconic search box for the first time in 25 years so that people could add photos and videos to their queries and assign A.I. “agents” to run searches for them.
The effect of these moves is becoming clear: People are spending more time with Google than ever…
… and spending less time with the sites that supply the information Google’s AI returns. Stephen Follows unpacks one web site’s experience– a cautionary tale for the open web and a reminder that, if we kill the sites generating the answers on which we depend, they won’t be there for anyone (directly or via AI) to find…
If you work in or around the film industry, there is a decent chance you have used the work of The Numbers this month, whether you realise it or not.
Its hand-researched data is the highest quality, tracking box office grosses, budgets, home video and streaming across more than 78,000 films and 236,000 people. It gets north of eight million visitors a year, and is treated as THE definitive authority by journalists, academics, filmmakers, prediction markets, and even Guinness World Records.
And it was this GOAT status which caused the catastrophic events of March this year.
The site was down for over a week, without explanation. A week later, it resurfaced at a fraction of its former size. Gone were the historical charts, the individual movie pages, and even the much-loved Report Builder.
With only a generic “we’re rebuilding, please bear with us” message to go on, the internet responded as it always does – with confusion, anger, and conspiracy theories. One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products.
Three months on, I spoke at length with Bruce Nash, founder and CEO of The Numbers, about what happened. He describes quite an unpleasant and eventful experience:
We got a lot of angry emails from people who are like, ‘Where’s this page that you used to have and you don’t have anymore?’
Within his tale are a number of things that should worry anyone who runs, relies on, or simply appreciates the internet…
[Follows recounts the history of The Numbers: its launch on Geocities, its growth, the onslaught of robots, the attack of hackers, and the coming of the LLMs and their relentless scraping…
… How bad could [AI bot scraping] be? Pretty bad, tbh. Enough that site owners such as Bruce have to question the value of something that will take so much time and money to build and defend.
Cloudflare, which protects a huge share of the world’s websites, publishes data on how many pages each AI platform crawls for every one visitor it sends back to the websites it crawled.
Google crawls about five pages for every visitor it sends you. OpenAI crawls over 1,000. Anthropic crawls over 38,000 pages for every single visitor it refers.
Note that the scale is logarithmic, i.e. each step along the bottom is ten times bigger than the last, because otherwise the differences are quite literally too large for me to include on one chart.
For the history of the internet to date, the principle of the open web was that, in return for letting the search engine robots read your site, they would send you readers. But now, that trade no longer applies. The number of robots has exploded, and they no longer send anyone back.
When this firehose is aimed at a small site, it can inflate the bandwidth bill and possibly even take down an entire site…
[Follows looks at other sites, including Wikipedia, in similar straights, and at attempts to protect these sites…]
… Whether any of this works depends on whether the AI companies play along rather than route around it. But as Bruce put it to me, somebody has to try.
Let’s look beyond the specifics for a moment and consider what happened here.
A beloved, useful, free website, run carefully by a competent, honest person for nearly thirty years, was crushed between two features of the new AI economy.
Unsustainable machine traffic hammered it from above, and in all likelihood a financially motivated intruder, operating in a world where breaking into websites has never been easier, took it down.
Bruce’s business and livelihood survived only because the website was not the whole business.
Others have not been so fortunate. Just last week, ZEGO, a German textile firm that had been in business for 37 years, filed for insolvency after a single cyberattack in March shut down its production for six weeks. Unlike The Numbers, they had no other business to fall back on.
The web is full of independent archives, hobby databases, local news sites, forums, reference works. Decades of accumulated human effort, running on old code, maintained by small teams or single individuals, quietly holding up far more of our shared knowledge than anyone acknowledges.
The Numbers is coming back, better built than before. I would encourage you to keep using it, keep supporting it, and, if you are one of the many people who emailed Bruce in fury about a missing page, perhaps send a kinder one now you know why it was missing…
As we conserve our culture, we might spare a thought for Vladimir Zworykin; he died on this date in 1982. An engineer and inventor, he is considered “the father of television” (or at least one of them). At the dawn of radio broadcasting, Zworykin began developing a system for transmitting sound and pictures. Other inventors were using a motorized, mechanical scanning system with rotating disks capable of a picture about one inch square– bulky, heavy, and impractical for home use. Zworykin, at Westinghouse, instead developed an electronic scanning television system using cathode-ray tubes; his innovations– among them, the iconoscope (the forerunner of the television camera) and the kinescope (which allowed the transfer of video to film)– laid the technical foundation for television as we came to know it… and contributed to the development of the electron microscope.
From yesterday’s post on the possible (and promising, but also potentially painful) future of computing to a pressing predicament we face today. The estimable Anil Dash on the threats to the open web…
You must imagine Sam Altman holding a knife to Tim Berners-Lee’s throat.
It’s not a pleasant image. Sir Tim is, rightly, revered as the genial father of the World Wide Web. But, all the signs are pointing to the fact that we might be in endgame for “open” as we’ve known it on the Internet over the last few decades.
The open web is something extraordinary: anybody can use whatever tools they have, to create content following publicly documented specifications, published using completely free and open platforms, and then share that work with anyone, anywhere in the world, without asking for permission from anyone. Think about how radical that is.
Now, from content to code, communities to culture, we can see example after example of that open web under attack. Every single aspect of the radical architecture I just described is threatened, by those who have profited most from that exact system.
Today, the good people who act as thoughtful stewards of the web infrastructure are still showing the same generosity of spirit that has created opportunity for billions of people and connected society in ways too vast to count while —not incidentally— also creating trillions of dollars of value and countless jobs around the world. But the increasingly-extremist tycoons of Big Tech have decided that that’s not good enough.
Now, the hectobillionaires have begun their final assault on the last, best parts of what’s still open, and likely won’t rest until they’ve either brought all of the independent and noncommercial parts of the Internet under their control, or destroyed them. Whether or not they succeed is going to be decided by decisions that we all make as a community in the coming months. Even though there have always been threats to openness on the web, the stakes have never been higher than they are this time.
Right now, too many of the players in the open ecosystem are still carrying on with business as usual, even though those tactics have been failing to stop big tech for years. I don’t say this lightly: it looks to me like 2026 is the year that decides whether the open web as we know it will survive at all, and we have to fight like the threat is existential. Because it is…
[Dash details the treats– largely, but not entirely driven by AI and its purveyors. He concludes…]
… The threat to the open web is far more profound than just some platforms that are under siege. The most egregious harm is the way that the generosity and grace of the people who keep the web open is being abused and exploited. Those people who maintain open source software? They’re hardly getting rich — that’s thankless, costly work, which they often choose instead of cashing in at some startup. Similarly, volunteering for Wikipedia is hardly profitable. Defining super-technical open standards takes time and patience, sometimes over a period of years, and there’s no fortune or fame in it.
Creators who fight hard to stay independent are often choosing to make less money, to go without winning awards or the other trappings of big media, just in order to maintain control and authority over their content, and because they think it’s the right way to connect with an audience. Publishers who’ve survived through year after year of attacks from tech platforms get rewarded by… getting to do it again the next year. Tim Berners-Lee is no billionaire, but none of those guys with the hundreds of billions of dollars would have all of their riches without him. And the thanks he gets from them is that they’re trying to kill the beautiful gift that he gave to the world, and replace it with a tedious, extortive slop mall.
So, we’re in endgame now. They see their chance to run the playbook again, and do to Wikipedians what Uber did to cab drivers, to get users addicted to closed apps like they are to social media, to force podcasters to chase an algorithm like kids on TikTok. If everyone across the open internet can gather together, and see that we’re all in one fight together, and push back with the same ferocity with which we’re being attacked, then we do have a shot at stopping them.
At one time, it was considered impossibly unlikely that anybody would ever create open technologies that would ever succeed in being useful for people, let alone that they would become a daily part of enabling billions of people to connect and communicate and make their lives better. So I don’t think it’s any more unlikely that the same communities can summon that kind of spirit again, and beat back the wealthiest people in the world, to ensure that the next generation gets to have these same amazing resources to rely on for decades to come.
Alright, if it’s not hopeless, what are the concrete things we can do? The first thing is to directly support organizations in the fight. Either those that are at risk, or those that are protecting those at risk. You can give directly to support the Internet Archive, or volunteer to help them out. Wikipedia welcomes your donation or your community participation. The Electronic Frontier Foundation is fighting for better policy and to defend your rights on virtually all of these issues, and could use your support or provides a list of ways to volunteer or take action. The Mozilla Foundation can also use your donations and is driving change. (And full disclosure — I’m involved in pretty much all of these organizations in some capacity, ranging from volunteer to advisor to board member.) That’s because I’m trying to make sure my deeds match my words! These are the people whom I’ve seen, with my own eyes, stay the hand of those who would hold the knife to the necks of the open web’s defenders. [Further full disclosure: so is your correpondent, and so have I.]
Beyond just what these organizations do, though, we can remember how much the open web matters. I know from my time on the board of Stack Overflow that we got to see the rise of an incredibly generous community built around sharing information openly, under open licenses. There are very few platforms in history that helped more people have more economic mobility than the number of people who got good-paying jobs as coders as a result of the information on that site. And then we got to see the toll that extractive LLMs had when they took advantage of that community without any consideration for the impact it would have when they trained models on the generosity of that site’s members without reciprocating in kind.
The good of the web only exists because of the openness of the web. They can’t just keep on taking and taking without expecting people to finally draw a line and saying “enough”. And interestingly, opportunities might exist where the tycoons least expect it. I saw Mike Masnick’s recent piece where he argued that one of the things that might enable a resurgence of the open web might be… AI. It would seem counterintuitive to anyone who’s read everything I’ve shared here to imagine that anything good could come of these same technologies that have caused so much harm.
But ultimately what matters is power. It is precisely because technologies like LLMs have powers that the authoritarians have rushed to try to take them over and wield them as effectively as they can. I don’t think that platforms owned and operated by those bad actors can be the tools that disrupt their agenda. I do think it might be possible that the creative communities that built the web in the first place could use their same innovative spirit to build what could be, for lack of a better term, called “good AI“. It’s going to take better policy, which may be impossible in the short term at the federal level in the U.S., but can certainly happen at more local levels and in the rest of the world. Though I’m skeptical about putting too much of the burden on individual users, we can certainly change culture and educate people so that more people feel empowered and motivated to choose alternatives to the big tech and big AI platforms that got us into this situation. And we can encourage harm reduction approaches for the people and institutions that are already locked into using these tools, because as we’ve seen, even small individual actions can get institutions to change course.
Ultimately I think, if given the choice, people will pick home-cooked, locally-grown, heart-felt digital meals over factory-farmed fast food technology every time…
As we protect what’s precious, we might send carefully-calculated birthday greetings to a man whose work helped lay the foundation for both the promise and the peril unpacked in the article linked above above: J. Presper Eckert; he was born on this day in 1919. An electrical engineer, he co-designed (with John Mauchly) the first general purpose computer, the ENIAC (see here and here) for the U.S. Army’s Ballistic Research Laboratory. He and Mauchy went on to found the Eckert–Mauchly Computer Corporation, at which they designed and built the first commercial computer in the U.S., the UNIVAC.
Eckert (standing and gesturing) and Mauchy (at the console), demonstrating the UNIVAC to Walter Cronkite (source)
You must be logged in to post a comment.