Posts Tagged ‘peer review’
“The pursuit of science is a grand adventure, driven by curiosity, fueled by passion, and guided by reason”*…
Adam Mastroianni on how science advances (and how it’s held back), with a provocative set of suggestions for how it might be accelerated…
There are two kinds of problems in the world: strong-link problems and weak-link problems.
Weak-link problems are problems where the overall quality depends on how good the worst stuff is. You fix weak-link problems by making the weakest links stronger, or by eliminating them entirely.
Food safety, for example, is a weak-link problem. You don’t want to eat anything that will kill you. That’s why it makes sense for the Food and Drug Administration to inspect processing plants, to set standards, and to ban dangerous foods…
…
Weak-link problems are everywhere. A car engine is a weak-link problem: it doesn’t matter how great your spark plugs are if your transmission is busted. Nuclear proliferation is a weak-link problem: it would be great if, say, France locked up their nukes even tighter, but the real danger is some rogue nation blowing up the world. Putting on too-tight pants is a weak-link problem: they’re gonna split at the seams.
It’s easy to assume that all problems are like this, but they’re not. Some problems are strong-link problems: overall quality depends on how good the best stuff is, and the bad stuff barely matters. Like music, for instance. You listen to the stuff you like the most and ignore the rest. When your favorite band releases a new album, you go “yippee!” When a band you’ve never heard of and wouldn’t like anyway releases a new album, you go…nothing at all, you don’t even know it’s happened. At worst, bad music makes it a little harder for you to find good music, or it annoys you by being played on the radio in the grocery store while you’re trying to buy your beetle-free asparagus…
Strong-link problems are everywhere; they’re just harder to spot. Winning the Olympics is a strong-link problem: all that matters is how good your country’s best athletes are. Friendships are a strong-link problem: you wouldn’t trade your ride-or-dies for better acquaintances. Venture capital is a strong-link problem: it’s fine to invest in a bunch of startups that go bust as long as one of them goes to a billion…
In the long run, the best stuff is basically all that matters, and the bad stuff doesn’t matter at all. The history of science is littered with the skulls of dead theories. No more phlogiston nor phlegm, no more luminiferous ether, no more geocentrism, no more measuring someone’s character by the bumps on their head, no more barnacles magically turning into geese, no more invisible rays shooting out of people’s eyes, no more plum pudding…
Our current scientific beliefs are not a random mix of the dumbest and smartest ideas from all of human history, and that’s because the smarter ideas stuck around while the dumber ones kind of went nowhere, on average—the hallmark of a strong-link problem. That doesn’t mean better ideas win immediately. Worse ideas can soak up resources and waste our time, and frauds can mislead us temporarily. It can take longer than a human lifetime to figure out which ideas are better, and sometimes progress only happens when old scientists die. But when a theory does a better job of explaining the world, it tends to stick around.
(Science being a strong-link problem doesn’t mean that science is currently strong. I think we’re still living in the Dark Ages, just less dark than before.)
Here’s the crazy thing: most people treat science like it’s a weak-link problem.
Peer reviewing publications and grant proposals, for example, is a massive weak-link intervention. We spend ~15,000 collective years of effort every year trying to prevent bad research from being published. We force scientists to spend huge chunks of time filling out grant applications—most of which will be unsuccessful—because we want to make sure we aren’t wasting our money…
…
I think there are two reasons why scientists act like science is a weak-link problem.
The first reason is fear. Competition for academic jobs, grants, and space in prestigious journals is more cutthroat than ever. When a single member of a grant panel, hiring committee, or editorial board can tank your career, you better stick to low-risk ideas. That’s fine when we’re trying to keep beetles out of asparagus, but it’s not fine when we’re trying to discover fundamental truths about the world…
The second reason is status. I’ve talked to a lot of folks since I published The rise and fall of peer review and got a lot of comments, and I’ve realized that when scientists tell me, “We need to prevent bad research from being published!” they often mean, “We need to prevent people from gaining academic status that they don’t deserve!” That is, to them, the problem with bad research isn’t really that it distorts the scientific record. The problem with bad research is that it’s cheating.
I get that. It’s maddening to watch someone get ahead using shady tactics, and it might seem like the solution is to tighten the rules so we catch more of the cheaters. But that’s weak-link thinking. The real solution is to care less about the hierarchy…
Here’s our reward for a generation of weak-link thinking.
The US government spends ~10x more on science today than it did in 1956, adjusted for inflation. We’ve got loads more scientists, and they publish way more papers. And yet science is less disruptive than ever, scientific productivity has been falling for decades, and scientists rate the discoveries of decades ago as worthier than the discoveries of today. (Reminder, if you want to blame this on ideas getting harder to find, I will fight you.)…
…
Whether we realize it or not, we’re always making calls like this. Whenever we demand certificates, credentials, inspections, professionalism, standards, and regulations, we are saying: “this is a weak-link problem; we must prevent the bad!”
Whenever we demand laissez-faire, the cutting of red tape, the letting of a thousand flowers bloom, we are saying: “this is a strong-link problem; we must promote the good!”
When we get this right, we fill the world with good things and rid the world of bad things. When we don’t, we end up stunting science for a generation. Or we end up eating a lot of asparagus beetles…
“Science is a strong-link problem,” from @a_m_mastroianni in @science_seeds.
* James Clerk Maxwell
###
As we ponder the process of progress, we might spare a thought for Sir Christopher Wren; he died on this date in 1723. A mathematician and astronomer (who co-founded and later served as president of the Royal Society), he is better remembered as one of the most highly acclaimed English architects in history; he was given responsibility for rebuilding 52 churches in the City of London after the Great Fire in 1666, including what is regarded as his masterpiece, St. Paul’s Cathedral, on Ludgate Hill.
Wren, whose scientific work ranged broadly– e.g., he invented a “weather clock” similar to a modern barometer, new engraving methods, and helped develop a blood transfusion technique– was admired by Isaac Newton, as Newton noted in the Principia.
“With my tongue in one cheek only, I’d suggest that had our palaeolithic ancestors discovered the peer-review dredger, we would be still sitting in caves”*…
As a format, “scholarly” scientific communications are slow, encourage hype, and are difficult to correct. Stuart Ritchie argues that a radical overhaul of publishing could make science better…
… Having been printed on paper since the very first scientific journal was inaugurated in 1665, the overwhelming majority of research is now submitted, reviewed and read online. During the pandemic, it was often devoured on social media, an essential part of the unfolding story of Covid-19. Hard copies of journals are increasingly viewed as curiosities – or not viewed at all.
But although the internet has transformed the way we read it, the overall system for how we publish science remains largely unchanged. We still have scientific papers; we still send them off to peer reviewers; we still have editors who give the ultimate thumbs up or down as to whether a paper is published in their journal.
This system comes with big problems. Chief among them is the issue of publication bias: reviewers and editors are more likely to give a scientific paper a good write-up and publish it in their journal if it reports positive or exciting results. So scientists go to great lengths to hype up their studies, lean on their analyses so they produce “better” results, and sometimes even commit fraud in order to impress those all-important gatekeepers. This drastically distorts our view of what really went on.
There are some possible fixes that change the way journals work. Maybe the decision to publish could be made based only on the methodology of a study, rather than on its results (this is already happening to a modest extent in a few journals). Maybe scientists could just publish all their research by default, and journals would curate, rather than decide, which results get out into the world. But maybe we could go a step further, and get rid of scientific papers altogether…
A bold proposal: “The big idea: should we get rid of the scientific paper?,” from @StuartJRitchie in @guardian.
Apposite (if only in its critical posture): “The Two Paper Rule.” See also “In what sense is the science of science a science?” for context.
###
As we noodle on knowledge, we might recall that it was on this date in 1964 that AT&T connected the first Picturephone call (between Disneyland in California and the World’s Fair in New York). The device consisted of a telephone handset and a small, matching TV, which allowed telephone users to see each other in fuzzy video images as they carried on a conversation. It was commercially-released shortly thereafter (prices ranged from $16 to $27 for a three-minute call between special booths AT&T set up in New York, Washington, and Chicago), but didn’t catch on.
“Peer review as practiced today is a form of hazing”*…
Cuneiform Letter from the astrologer Marduk-šapik-zeri to the Neo-Assyrian king Esarhaddon
The advance of science depends on the communications of research and experimental findings so that they can be, first, replicated and verified or refuted; then broadly understood by the scientific community. Historically, that communication has depended largely on scientific journals, the primary vehicles of that dissemination. The integrity of the system has depended on the peer-review process: the examination of scientific papers submitted for journal publication by a jury of “peers” (in practice, usually very senior practitioners of the discipline in question) who evaluate the methodology and findings being reported and pass on whether or not they are “publishable.”
With the advent of the web, this system is loosening. Scientists are sharing “pre-prints” in sites like arXiv, reaching around the journals’ referees to reach their communities at large. Still, the feedback that they get is a form of peer review…
While we tend to date the birth of the scientific method, and this approach, to the early 17th century and the thinking of Bacon and Descartes, archaeologists suggest that the approach might have have much deeper roots…
In some respects, the life of a Mesopotamian scholar in the seventh century B.C. was not so very different from that of a modern academic. While the former might be responsible for reporting on celestial phenomena and whether they augur well for the king’s reign, and the latter might be searching for evidence of a new subatomic particle to better understand the origins of the universe, in either case, one’s reputation among colleagues is paramount.
Let’s take, for example, the lot of an unnamed astrologer who was subjected to a vicious onslaught of peer review from some of the Neo-Assyrian Empire’s top minds after claiming to have sighted Venus around 669 B.C. In a letter to the king Esarhaddon (r. 680–669 B.C.), a fellow stargazer named Nabû-ahhe-eriba, who was part of the inner circle of royal scholars, inveighed, “(He who) wrote to the king, my lord, ‘The planet Venus is visible, it is visible (in the month Ad)ar,’ is a vile man, an ignoramus, a cheat!” Slightly more charitable, though still cutting, was a scholar named Balasî, who tutored the crown prince Ashurbanipal (r. 668–627 B.C.). “(T)he man who wrote (thus) to the king, (my lord), is in ignorance,” Balasî informed Esarhaddon. “The ig(noramus)—who is he?…I repeat: He does not understand (the difference) between Mercury and Venus.”
These quotations are excerpts from just two of around 1,000 letters and reports written by scholars to Esarhaddon and Ashurbanipal in cuneiform on clay tablets that were discovered during nineteenth-century excavations of the archives of the Assyrian capital, Nineveh, near Mosul in Iraq, including Ashurbanipal’s library…
The perils of peer review– what was old is new again: “Ancient academia.”
###
As we contemplate constructive criticism, we might send repetitious birthday greetings to Émile Coué de la Châtaigneraie; he was born on this date in 1857. A pharmacist who began practicing as a psychologist, Coué opened a clinic in Nancy, and introduced a method of psychotherapy characterized by frequent repetition of the formula, je vais de mieux en mieux, “Every day, and in every way, I am becoming better and better”; he counseled his patients to repeat this 15 to 20 times, morning and evening. This method of autosuggestion came to be called Couéism, and was very popular in the 1920s and 1930s. (Norman Vincent Peale’s brand of positive thinking was rooted in part in Coué’s work.) The popular press raved about his approach, even as the medical and psychological establishment dismissed it. And as the seemingly positive results he achieved with his patients faded– as they seemed for the most part to do– so did enthusiasm for the Coué method. Still, one can hear its echo in approaches alive today, for instance neuro-linguistic programming.
A contemporary, Rev. Charles Inge, captured Coué’s simplistic method in a limerick (1928): “This very remarkable man / Commends a most practical plan: / You can do what you want / If you don’t think you can’t, / So don’t think you can’t think you can.”
You must be logged in to post a comment.