Times have changed. As the New York Times reports (in a piece apposite to Tuesday’s post)…
When Google prepared to go public in 2004, Larry Page, a co-founder of the company, wrote a letter to shareholders describing the internet firm’s responsibility to the world.
“We believe a well-functioning society should have abundant, free and unbiased access to high-quality information,” Mr. Page said.
Google fulfilled that responsibility by acting as a gateway to the internet. It answered people’s search queries with lists of hyperlinks, pushing users out to what is known as the “open web” — the millions of websites run by merchants, publishers, universities and others — for more information. In the ensuing decades, Google became one of the planet’s richest and most powerful companies by directing people to the vastness of the open web.
Now in the age of artificial intelligence, Google appears to be shrinking back from the open web — and may be imperiling it.
Since last year, the Silicon Valley giant has revamped its search with A.I. It introduced AI Mode, which replaces search results of hyperlinks with conversational responses written by Gemini, its A.I. chatbot. Most recently, Google changed its iconic search box for the first time in 25 years so that people could add photos and videos to their queries and assign A.I. “agents” to run searches for them.
The effect of these moves is becoming clear: People are spending more time with Google than ever…
… and spending less time with the sites that supply the information Google’s AI returns. Stephen Follows unpacks one web site’s experience– a cautionary tale for the open web and a reminder that, if we kill the sites generating the answers on which we depend, they won’t be there for anyone (directly or via AI) to find…
If you work in or around the film industry, there is a decent chance you have used the work of The Numbers this month, whether you realise it or not.
Its hand-researched data is the highest quality, tracking box office grosses, budgets, home video and streaming across more than 78,000 films and 236,000 people. It gets north of eight million visitors a year, and is treated as THE definitive authority by journalists, academics, filmmakers, prediction markets, and even Guinness World Records.
And it was this GOAT status which caused the catastrophic events of March this year.
The site was down for over a week, without explanation. A week later, it resurfaced at a fraction of its former size. Gone were the historical charts, the individual movie pages, and even the much-loved Report Builder.
With only a generic “we’re rebuilding, please bear with us” message to go on, the internet responded as it always does – with confusion, anger, and conspiracy theories. One Reddit theory even suggested it was a deliberate rug pull designed to cripple the free site to push people towards paid products.
Three months on, I spoke at length with Bruce Nash, founder and CEO of The Numbers, about what happened. He describes quite an unpleasant and eventful experience:
We got a lot of angry emails from people who are like, ‘Where’s this page that you used to have and you don’t have anymore?’
Within his tale are a number of things that should worry anyone who runs, relies on, or simply appreciates the internet…
[Follows recounts the history of The Numbers: its launch on Geocities, its growth, the onslaught of robots, the attack of hackers, and the coming of the LLMs and their relentless scraping…
… How bad could [AI bot scraping] be? Pretty bad, tbh. Enough that site owners such as Bruce have to question the value of something that will take so much time and money to build and defend.
Cloudflare, which protects a huge share of the world’s websites, publishes data on how many pages each AI platform crawls for every one visitor it sends back to the websites it crawled.
Google crawls about five pages for every visitor it sends you. OpenAI crawls over 1,000. Anthropic crawls over 38,000 pages for every single visitor it refers.
Note that the scale is logarithmic, i.e. each step along the bottom is ten times bigger than the last, because otherwise the differences are quite literally too large for me to include on one chart.
For the history of the internet to date, the principle of the open web was that, in return for letting the search engine robots read your site, they would send you readers. But now, that trade no longer applies. The number of robots has exploded, and they no longer send anyone back.
When this firehose is aimed at a small site, it can inflate the bandwidth bill and possibly even take down an entire site…
[Follows looks at other sites, including Wikipedia, in similar straights, and at attempts to protect these sites…]
… Whether any of this works depends on whether the AI companies play along rather than route around it. But as Bruce put it to me, somebody has to try.
Let’s look beyond the specifics for a moment and consider what happened here.
A beloved, useful, free website, run carefully by a competent, honest person for nearly thirty years, was crushed between two features of the new AI economy.
Unsustainable machine traffic hammered it from above, and in all likelihood a financially motivated intruder, operating in a world where breaking into websites has never been easier, took it down.
Bruce’s business and livelihood survived only because the website was not the whole business.
Others have not been so fortunate. Just last week, ZEGO, a German textile firm that had been in business for 37 years, filed for insolvency after a single cyberattack in March shut down its production for six weeks. Unlike The Numbers, they had no other business to fall back on.
The web is full of independent archives, hobby databases, local news sites, forums, reference works. Decades of accumulated human effort, running on old code, maintained by small teams or single individuals, quietly holding up far more of our shared knowledge than anyone acknowledges.
The Numbers is coming back, better built than before. I would encourage you to keep using it, keep supporting it, and, if you are one of the many people who emailed Bruce in fury about a missing page, perhaps send a kinder one now you know why it was missing…
As we conserve our culture, we might spare a thought for Vladimir Zworykin; he died on this date in 1982. An engineer and inventor, he is considered “the father of television” (or at least one of them). At the dawn of radio broadcasting, Zworykin began developing a system for transmitting sound and pictures. Other inventors were using a motorized, mechanical scanning system with rotating disks capable of a picture about one inch square– bulky, heavy, and impractical for home use. Zworykin, at Westinghouse, instead developed an electronic scanning television system using cathode-ray tubes; his innovations– among them, the iconoscope (the forerunner of the television camera) and the kinescope (which allowed the transfer of video to film)– laid the technical foundation for television as we came to know it… and contributed to the development of the electron microscope.
You must be logged in to post a comment.