(Roughly) Daily

Posts Tagged ‘scientific method

Playing the odds…

A P value is the probability of an observed (or more extreme) result arising only from chance.

It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.

Replicating a result helps establish its validity more securely, but the common tactic of combining numerous studies into one analysis, while sound in principle, is seldom conducted properly in practice.

Experts in the math of probability and statistics are well aware of these problems and have for decades expressed concern about them in major journals. Over the years, hundreds of published papers have warned that science’s love affair with statistics has spawned countless illegitimate findings. In fact, if you believe what you read in the scientific literature, you shouldn’t believe what you read in the scientific literature.

“There is increasing concern,” declared epidemiologist John Ioannidis in a highly cited 2005 paper in PLoS Medicine, “that in modern research, false findings may be the majority or even the vast majority of published research claims.”

Ioannidis claimed to prove that more than half of published findings are false, but his analysis came under fire for statistical shortcomings of its own. “It may be true, but he didn’t prove it,” says biostatistician Steven Goodman of the Johns Hopkins University School of Public Health. On the other hand, says Goodman, the basic message stands. “There are more false claims made in the medical literature than anybody appreciates,” he says. “There’s no question about that.”

Nobody contends that all of science is wrong, or that it hasn’t compiled an impressive array of truths about the natural world. Still, any single scientific study alone is quite likely to be incorrect, thanks largely to the fact that the standard statistical system for drawing conclusions is, in essence, illogical. “A lot of scientists don’t understand statistics,” says Goodman. “And they don’t understand statistics because the statistics don’t make sense”…

What’s one to make of the stream of “eat this,” “avoid that” studies surfacing nearly daily?  It’s an odds-on bet that readers will find out in the complete Science News story, “Odds Are, It’s Wrong.”

As we tell Monty that we’ll take what’s behind Door #2, we might recall that it was on this date in 1905 that Albert Einstein kicked off  “Annus Mirabilis” with the publication of the first of his four epoch-making papers in Annalen der Physik— this one, proposing energy “quanta”– thus kicking off the year in which he reinvented physics and our understanding of reality.

The second of those papers, on Brownian motion, was the very first work of “statistical physics.”

Einstein, dressed for the patent office, 1905

Happy Náw-Rúz! This date in 1844 was the first day of the first year of the Bahai calendar.

Management tools for tough times…

A manager’s lot has always been a hard one; but in these troubled times, aggravated by the challenge of motivating a work force that’s fearful and confused, it’s downright daunting.

So praise be to the good folks at Sloshspot (“for partiers, bar-goers, bar-owners, bartenders, and anyone who likes to party”)– they’ve just made available Hunter S. Thompson Motivational Posters.

See them all here.

As we get in touch with our inner gonzo, we might light a birthday candle for Sir Francis Bacon– English Renaissance philosopher, lawyer, linguist, composer, mathematician, geometer, musician, poet, painter, astronomer, classicist, philosopher, historian, theologian, architect, father of modern science (The Baconian– aka The Scientific– Method), and patron of modern democracy, whom some allege was the illegitimate son of Queen Elizabeth I of England… but who was in any event born on this date in 1561.

Bacon (whose Essays were, in a fashion, the first “management book” in English) was, in Alexander Pope’s words, “the greatest genius that England, or perhaps any country, ever produced.”  He probably did not actually write the plays attributed to Shakespeare (as a thin, but long, line of scholars and observers, including Mark Twain and Friedrich Nietzsche, believed).   But Bacon did observe, in a discussion of sedition that’s as timely today as ever, that “the remedy is worse than the disease.”

Sir Francis Bacon

%d bloggers like this: