Posts Tagged ‘statistics’
“The generation of random numbers is too important to be left to chance”*…
Roman 12mm dice
Random numbers are central to more than we may realize. They have applications in gambling, statistical sampling, computer simulation and Monte Carlo modeling, cryptography (as applied in both communications and transactions), completely randomized design, even sooth-saying– in any area where producing an unpredictable result is desirable. So how they’re produced– the certainty that they are, in fact, random– matters enormously.
It’s no surprise, then, that random number generation has a long and fascinating history. Happily, Carl Tashian is here to explain.
“As an instrument for selecting at random, I have found nothing superior to dice,” wrote statistician Francis Galton in an 1890 issue of Nature. “When they are shaken and tossed in a basket, they hurtle so variously against one another and against the ribs of the basket-work that they tumble wildly about, and their positions at the outset afford no perceptible clue to what they will be even after a single good shake and toss.”…
From I Ching sticks and dice to the cryptographically-secure PRNG, “A Brief History of Random Numbers.”
[TotH to the eminently-numerate Reuben Steiger]
###
As we roll the bones, we might spare a thought for Samuel “Sam” Loyd; he died on this date in 1911. A chess player, chess composer, puzzle author, and recreational mathematician. A member of the Chess Hall of Fame (for both his play and for his exercises, or “problems”), he gained posthumous fame when his son published a collection of his mathematical and logic puzzles, Cyclopedia of 5000 Puzzles after his father’s death. As readers can see here and here, his puzzles still delight.
Loyd’s most famous puzzle was the 14-15 Puzzle, which he produced in 1878. His original authorship is debated; but in any case, his version created a craze that swept America to such an extent that employers put up notices prohibiting playing the puzzle during office hours.
“Without data, you’re just another person with an opinion”*…
It used to be that we’d see a poorly made graph or a data design goof, laugh it up a bit, and then carry on. At some point though — during this past year especially — it grew more difficult to distinguish a visualization snafu from bias and deliberate misinformation.
Of course, lying with statistics has been a thing for a long time, but charts tend to spread far and wide these days. There’s a lot of them. Some don’t tell the truth. Maybe you glance at it and that’s it, but a simple message sticks and builds. Before you know it, Leonardo DiCaprio spins a top on a table and no one cares if it falls or continues to rotate.
So it’s all the more important now to quickly decide if a graph is telling the truth…
Nathan Yau (Flowing Data) provides a very helpful (and very amusing) guide: “How to Spot Visualization Lies.”
* W. Edwards Deming
###
As we key our eyes open, we might send healthy birthday greetings to John Snow; he was born on this date in 1813. A physician and a leader in the adoption of anaesthesia and medical hygiene, he is considered the father of modern epidemiology, in large measure because of his work in tracing the source of a cholera outbreak in Soho, London, in 1854. His On the Mode of Communication of Cholera (1849) suggested that cholera was a contagious disease easily transmitted by contaminated water. But the widely-held theory was that diseases are caused by bad air led to his idea being ignored. Then, in London’s 1854 cholera emergency, he painstakingly correlated individual cholera casualties to the water supply they had used in each case. He then communicated his results with a map that underlined his point, and ended the deadly epidemic by removing the pump handle of the community water pump that he found to be the culprit.
Snow’s map of cholera cases
His findings inspired fundamental changes in the water and waste systems of London, which led to similar changes in other cities, and a significant improvement in general public health around the world. His mode of communicating them contributed to the rise of data visualization.
“When you have mastered numbers, you will in fact no longer be reading numbers, any more than you read words when reading books. You will be reading meanings.”*…
Errors of judgment about large numbers can have a big impact on the way you view policies and government decisions. The rationale goes like this: The National Science Foundation received $7.463 billion for fiscal year 2016 through the Consolidated Appropriations Act. The total United States budget outlay for 2016 was $3.54 trillion. If you’re someone who perceives the difference between a billion and a trillion as relatively small, you’d think the US is spending a lot of money on the National Science Foundation—in fact, depending on your politics, you might applaud the federal government’s investment or even think it wasteful. But, if you understand that a billion is a thousand times less than a trillion, you can calculate that the Foundation got a paltry 0.2 percent of the budget outlay last year. (It may be more straightforward to think of the budget as roughly one-half to one-third of reported costs for the proposed US-Mexico border wall, and let your values guide you from there.)…
On the significance of scale: “How to Understand Extreme Numbers.”
[The image above is, of course, from the ever-wonderful xkcd.]
* W.E.B. Du Bois
###
As we nudge ourselves toward numeracy, we might spare a thought for Sewall Wright; he died on this date in 1988. A geneticist, he was known for his influential work on evolutionary theory and also for his work on path analysis. He was a founder (with Ronald Fisher and J.B.S. Haldane) of population genetics– a major step in the development of the modern evolutionary synthesis combining genetics with evolution. He is perhaps best remembered for his concept of genetic drift (called the Sewall Wright effect): when small populations of a species are isolated, the few individuals who carry certain relatively rare genes may fail, out of pure chance, to transmit them. The genes may therefore disappear and their loss may lead to the emergence of new species– although natural selection has played no part in the process.
“A certain elementary training in statistical method is becoming as necessary for everyone living in this world of today as reading and writing”*…
The declining authority of statistics – and the experts who analyse them – is at the heart of the crisis that has become known as “post-truth” politics. And in this uncertain new world, attitudes towards quantitative expertise have become increasingly divided. From one perspective, grounding politics in statistics is elitist, undemocratic and oblivious to people’s emotional investments in their community and nation. It is just one more way that privileged people in London, Washington DC or Brussels seek to impose their worldview on everybody else. From the opposite perspective, statistics are quite the opposite of elitist. They enable journalists, citizens and politicians to discuss society as a whole, not on the basis of anecdote, sentiment or prejudice, but in ways that can be validated. The alternative to quantitative expertise is less likely to be democracy than an unleashing of tabloid editors and demagogues to provide their own “truth” of what is going on across society.
Is there a way out of this polarisation? Must we simply choose between a politics of facts and one of emotions, or is there another way of looking at this situation?One way is to view statistics through the lens of their history. We need to try and see them for what they are: neither unquestionable truths nor elite conspiracies, but rather as tools designed to simplify the job of government, for better or worse. Viewed historically, we can see what a crucial role statistics have played in our understanding of nation states and their progress. This raises the alarming question of how – if at all – we will continue to have common ideas of society and collective progress, should statistics fall by the wayside…
The ability of statistics to represent the world accurately is declining. In its wake, a new age of big data controlled by private companies is taking over – and putting democracy in peril. William Davies provides historical context, a clear diagnosis of the problem, and thoughts on a response in his important essay, “How statistics lost their power – and why we should fear what comes next.”
* H.G. Wells, World Brain (1938)
###
As we take note of numbers, we might send insightful birthday greetings to Roger Newland Shepard; he was born on this date in 1929. A cognitive scientist and emeritus professor at Stanford, he has received both the National Medal of Science and the Rumelhart Prize. While his contributions to his field are many, Shepard is probably best known as inventor of multidimensional scaling, a method for representing certain kinds of statistical data in a plane (or in space) with minimal distortion, so that the data can be apprehended by non-specialists.
“I go to seek a Great Perhaps”*…
As we’ve noted before, 2016 seemed a bumper year for the Grim Reaper. Jason Crease tested that perception against the data…
It’s become cliché that unusually many prominent people died in 2016. Is this true?…
Find out at “Was 2016 especially dangerous for celebrities? An empirical analysis.“
[Image above: source]
* François Rabelais
###
As we usher in the new, we might spare a thought for the first woman in the Western world considered to be a mathematician: Maria Gaetana Agnesi, she died this date in 1799. While she thought and wrote broadly about natural science and philosophy, she is best remembered for her work in differential calculus– perhaps most particularly for her work on the cubic curve now know as the “witch of Agnesi.”
“Human progress is neither automatic nor inevitable”*…
… but happily, progress is made.
As we fight hangovers (both from New Years festivities and from the slow-motion train wreck that was 2016), Max Roser reminds us that in many critical dimensions life has gotten better- much better– around the world… and he reminds us why it’s so very important that we understand this…
“A history of global living conditions in 5 charts.”
* “Human progress is neither automatic nor inevitable… Every step toward the goal of justice requires sacrifice, suffering, and struggle; the tireless exertions and passionate concern of dedicated individuals.” ― Martin Luther King, Jr.
###
As we look on the bright side, we might that it was on this date in 1890 that E.A. McIlhenny, the son of Tabasco brand pepper sauce inventor Edmund McIlhenny and manager of the family condiment empire, shot and killed a 19′ 1″ long alligator, reputedly the longest American alligator ever recorded. McIlhenny, who was an amateur naturalist and conservationist, made the claim in one of his four books, The Alligator’s Life History.
“There is nothing more deceptive than an obvious fact”*…
Your correspondent is heading into his annual Holiday hiatus (regular service will resume on January 2), so by way of being sure that readers have something to occupy them in the meantime…
We’ve spent time before with Hans Rosling (“By the Numbers…” and “Poverty is an anomaly to rich people. It is very difficult to make out why people who want dinner do not ring the bell“*…), the man who reasons that experts cannot solve major challenges if they do not operate on facts. “But first you need to erase preconceived ideas,” he says, “and that is the difficult thing.” Nature has done us all the service of collecting a wide variety of his amazing, mind-changing presentations…
Hans Rosling knew never to flee from men wielding machetes. “The risk is higher if you run than if you face them,” he says. So, in 1989, when an angry mob confronted him at the field laboratory he had set up in what is now the Democratic Republic of the Congo, Rosling tried to appear calm. “I thought, ‘I need to use the resources I have, and I am good at talking’.”
Rosling, a physician and epidemiologist, pulled from his knapsack a handful of photographs of people from different parts of Africa who had been crippled by konzo, an incurable disease that was affecting many in this community, too. Through an interpreter, he explained that he believed he knew the cause, and he wanted to test local people’s blood to be sure. A few minutes into his demonstration, an old woman stepped forward and addressed the crowd in support of the research. After the more aggressive members of the mob stopped waving their machetes, she rolled up her sleeve. Most followed her lead. “You can do anything as long as you talk with people and listen to people and talk with the intelligentsia of the community,” says Rosling.
He is still trying to arm influential people with facts. He has become a trusted counsellor and speaker of plain truth to United Nations leaders, billionaire executives such as Facebook’s Mark Zuckerberg and politicians including Al Gore. Even Fidel Castro called on the slim, bespectacled Swede for advice. Rosling’s video lectures on global health and economics have elevated him to viral celebrity status, and he has been listed among the 100 most influential people in the world by the magazines Time and Foreign Policy. Melinda Gates of the Bill & Melinda Gates Foundation says, “To have Hans Rosling as a teacher is one of the biggest honors in the world.”…
More of Rosling’s story– and lots of his short-but-world-view-changing presentations– at “Three minutes with Hans Rosling will change your mind about the world.”
* Arthur Conan Doyle, “The Boscombe Valley Mystery”
###
As we celebrate facts, we might send amazing birthday greetings to John Nevil Maskelyne; he was born on this date in 1839. An accomplished stage magician, he was also an inventor, perhaps most notably, of the first pay toilet (the “penny toilet,” origin of the phrase “spend a penny”).
His son, Nevil Maskelyne, was also a magician and inventor– and the perpetrator of the first instance of electronic hacking.
John Nevil Maskelyne
Happy Holidays! See you in the New Year…
