Posts Tagged ‘peer review’
“With my tongue in one cheek only, I’d suggest that had our palaeolithic ancestors discovered the peer-review dredger, we would be still sitting in caves”*…
As a format, “scholarly” scientific communications are slow, encourage hype, and are difficult to correct. Stuart Ritchie argues that a radical overhaul of publishing could make science better…
… Having been printed on paper since the very first scientific journal was inaugurated in 1665, the overwhelming majority of research is now submitted, reviewed and read online. During the pandemic, it was often devoured on social media, an essential part of the unfolding story of Covid-19. Hard copies of journals are increasingly viewed as curiosities – or not viewed at all.
But although the internet has transformed the way we read it, the overall system for how we publish science remains largely unchanged. We still have scientific papers; we still send them off to peer reviewers; we still have editors who give the ultimate thumbs up or down as to whether a paper is published in their journal.
This system comes with big problems. Chief among them is the issue of publication bias: reviewers and editors are more likely to give a scientific paper a good write-up and publish it in their journal if it reports positive or exciting results. So scientists go to great lengths to hype up their studies, lean on their analyses so they produce “better” results, and sometimes even commit fraud in order to impress those all-important gatekeepers. This drastically distorts our view of what really went on.
There are some possible fixes that change the way journals work. Maybe the decision to publish could be made based only on the methodology of a study, rather than on its results (this is already happening to a modest extent in a few journals). Maybe scientists could just publish all their research by default, and journals would curate, rather than decide, which results get out into the world. But maybe we could go a step further, and get rid of scientific papers altogether…
A bold proposal: “The big idea: should we get rid of the scientific paper?,” from @StuartJRitchie in @guardian.
Apposite (if only in its critical posture): “The Two Paper Rule.” See also “In what sense is the science of science a science?” for context.
###
As we noodle on knowledge, we might recall that it was on this date in 1964 that AT&T connected the first Picturephone call (between Disneyland in California and the World’s Fair in New York). The device consisted of a telephone handset and a small, matching TV, which allowed telephone users to see each other in fuzzy video images as they carried on a conversation. It was commercially-released shortly thereafter (prices ranged from $16 to $27 for a three-minute call between special booths AT&T set up in New York, Washington, and Chicago), but didn’t catch on.
“Peer review as practiced today is a form of hazing”*…
Cuneiform Letter from the astrologer Marduk-šapik-zeri to the Neo-Assyrian king Esarhaddon
The advance of science depends on the communications of research and experimental findings so that they can be, first, replicated and verified or refuted; then broadly understood by the scientific community. Historically, that communication has depended largely on scientific journals, the primary vehicles of that dissemination. The integrity of the system has depended on the peer-review process: the examination of scientific papers submitted for journal publication by a jury of “peers” (in practice, usually very senior practitioners of the discipline in question) who evaluate the methodology and findings being reported and pass on whether or not they are “publishable.”
With the advent of the web, this system is loosening. Scientists are sharing “pre-prints” in sites like arXiv, reaching around the journals’ referees to reach their communities at large. Still, the feedback that they get is a form of peer review…
While we tend to date the birth of the scientific method, and this approach, to the early 17th century and the thinking of Bacon and Descartes, archaeologists suggest that the approach might have have much deeper roots…
In some respects, the life of a Mesopotamian scholar in the seventh century B.C. was not so very different from that of a modern academic. While the former might be responsible for reporting on celestial phenomena and whether they augur well for the king’s reign, and the latter might be searching for evidence of a new subatomic particle to better understand the origins of the universe, in either case, one’s reputation among colleagues is paramount.
Let’s take, for example, the lot of an unnamed astrologer who was subjected to a vicious onslaught of peer review from some of the Neo-Assyrian Empire’s top minds after claiming to have sighted Venus around 669 B.C. In a letter to the king Esarhaddon (r. 680–669 B.C.), a fellow stargazer named Nabû-ahhe-eriba, who was part of the inner circle of royal scholars, inveighed, “(He who) wrote to the king, my lord, ‘The planet Venus is visible, it is visible (in the month Ad)ar,’ is a vile man, an ignoramus, a cheat!” Slightly more charitable, though still cutting, was a scholar named Balasî, who tutored the crown prince Ashurbanipal (r. 668–627 B.C.). “(T)he man who wrote (thus) to the king, (my lord), is in ignorance,” Balasî informed Esarhaddon. “The ig(noramus)—who is he?…I repeat: He does not understand (the difference) between Mercury and Venus.”
These quotations are excerpts from just two of around 1,000 letters and reports written by scholars to Esarhaddon and Ashurbanipal in cuneiform on clay tablets that were discovered during nineteenth-century excavations of the archives of the Assyrian capital, Nineveh, near Mosul in Iraq, including Ashurbanipal’s library…
The perils of peer review– what was old is new again: “Ancient academia.”
###
As we contemplate constructive criticism, we might send repetitious birthday greetings to Émile Coué de la Châtaigneraie; he was born on this date in 1857. A pharmacist who began practicing as a psychologist, Coué opened a clinic in Nancy, and introduced a method of psychotherapy characterized by frequent repetition of the formula, je vais de mieux en mieux, “Every day, and in every way, I am becoming better and better”; he counseled his patients to repeat this 15 to 20 times, morning and evening. This method of autosuggestion came to be called Couéism, and was very popular in the 1920s and 1930s. (Norman Vincent Peale’s brand of positive thinking was rooted in part in Coué’s work.) The popular press raved about his approach, even as the medical and psychological establishment dismissed it. And as the seemingly positive results he achieved with his patients faded– as they seemed for the most part to do– so did enthusiasm for the Coué method. Still, one can hear its echo in approaches alive today, for instance neuro-linguistic programming.
A contemporary, Rev. Charles Inge, captured Coué’s simplistic method in a limerick (1928): “This very remarkable man / Commends a most practical plan: / You can do what you want / If you don’t think you can’t, / So don’t think you can’t think you can.”
You must be logged in to post a comment.