| Friday, September
14, 2007
Subj: Why Most Published
Research Findings Are False
Recall the controversy about the extent to which the
research supporting the theory of Anthropogenic Global Warming can be
considered scientific, since the researchers refuse to disclose sufficient
information about their materials and methods to permit independent
replication.
There was an article -- unfortunately available only
to subscribers -- in the _Wall Street Journal_ today about John P. A.
Ioannidis, the author of the following piece, which the article says
"remains the most downloaded technical paper that the journal PLoS Medicine
has ever published":
http://dx.doi.org/10.1371/journal.pmed.0020124
>>There is increasing concern that most current
published research findings are false. The probability that a research claim
is true may depend on study power and bias, the number of other studies on
the same question, and, importantly, the ratio of true to no relationships
among the relationships probed in each scientific field. In this framework,
a research finding is less likely to be true when the studies conducted in a
field are smaller; when effect sizes are smaller; when there is a greater
number and lesser preselection of tested relationships; where there is
greater flexibility in designs, definitions, outcomes, and analytical modes;
when there is greater financial and other interest and prejudice; and when
more teams are involved in a scientific field in chase of statistical
significance. Simulations show that for most study designs and settings, it
is more likely for a research claim to be false than true. Moreover, for
many current scientific fields, claimed research findings may often be
simply accurate measures of the prevailing bias. In this essay, I discuss
the implications of these problems for the conduct and interpretation of
research.<<
This is interesting for at least two reasons: (1) the
matter itself; (2) an example of "no permission required" publication.
The WSJ article also linked to this:
Most Published Research Findings Are False—But a
Little Replication Goes a Long Way
http://dx.doi.org/10.1371/journal.pmed.0040028
This leads me back to something I've been wondering
for several years: should part of every PhD -- or even Master's -- program
be *replicating* a published result, or trying to? Or should special
recognition be given to *replicated* results? There used to be a _Journal of
Irreproducible Results_, which was kind of a joke; maybe it was a more
ironic joke than we knew, and what we need is a _Journal of Reproducible
Results_?
It is my understanding that the guidance for the
bodies allocating public funds for research grants excludes funding for
research that merely replicates previous experiments. Should some fraction
of public research funding be set aside for replicating (or not) the
supposed new discoveries of publicly funded research?
The late statistician John Tukey, of Princeton and
Bell Labs, used to distinguish between
Exploration -- looking for indications that something
is going on,
Confirmation -- looking for evidence about whether
previously-noticed indications are spurious or not -- and
Adjudication -- weighing the all the evidence to make
a judgment about whether, in what sense, and to what extent, something has
been "proved".
Do we need more explicit support, both financial and
social, for confirmatory and adjudicatory research?
Peter Drucker used to say that making an effective
decision has to start with *disagreement*. Do we need more disagreement in
science?
Investigations by the Catholic Church of candidates
for sainthood include participation by an officer known popularly as the "Advocatus
Diaboli" -- the Devil's Advocate -- whose job is to marshal all arguments
*against* the candidate. Do we need such officers in our scientific
communities, or at least in the organs of our government that try to use the
results of scientific research to guide public policy?
How can we make sure that *negative* results get
published, so they're available to be weighed in the balance against the
"positive" results of similar studies?
And finally: where are the science-fiction stories
that might provide cautionary tales to inoculate youngsters who'll
eventually be the next generation of scientists, maybe even of
science-policy makers? I see _Fallen Angels_; what else is there?
Rod
Montgomery==monty@starfief.com
I'm working on them. But I have to make a
living and that kind of story doesn't sell as well as big thrillers. Now if
we had a few thousand subscribers...
The institution of a Devil's Advocate in
our scientific deliberations would be a good thing; it would at least put
one different view onto our increasingly monolithic campuses. Of course we
need more disagreement in science; but the politicization of science has
gone far. Note that we still have put all our AIDS research investments in
the HIV hypothesis without any crucial experiments. Perhaps that is wise;
but it still seems to me that a few million in contrarian research would be
a good thing before we discard Koch's Criteria. In this case we know who
ought to be funded, too.
I spent a wonderful weekend with Tukey
while working on a project for NASA. Minsky was there as well (I roomed with
Minsky). It was one of my better weekends.
==
Well, just to play "Devils Advocate" :-)
I have a problem with an effort devoted to replicating
results. That addresses one as of the issue--identifying faulty, erroneous,
or created data. It does not address the issue of poor experiment design.
From another point of view, though, perhaps what you
suggest is analogous to an Audit. In an audit, done by CPAs, internal
controls are review of internal controls (analyzing the structures and
processes that gather and organize the data into financial statements-) in
order to determine how much reliance you can place on the accurate recording
of data, and then testing transactions (individual data points). More
reliable structures and systems result in less transactional testing. And
the auditors write Management Letters identifying weaknesses in internal
controls and recommending solutions to fix those weaknesses.
Should some fraction of each research grant be set
aside for such audits ? Who is capable and interested in such work. It seems
like it would be a paid role, not an unpaid peer review role....
Could there not be standards created for experiment
design, like there are for bridge design (oops, perhaps that's a bad analogy
given recent events in Minneapolis) or manufacture of goods (oops, another
bad analogy, given the recent discovery of lead paint in toys sold by
Mattel) ? Well, thinking it through, perhaps standards address many, but not
all issues. But standards can help reduce, but perhaps not eliminate, errors
in design.
Still, it seems to me to be a quality control issue,
addressable at least in part by applying some of Dr. Deming's thoughts to
the research domain.
Dave Montgomery
I would say it's pretty clear that we
ought routinely to fund contrarian research.
=========
The following exchange took place
in another conference. Chris Brand's response is posted with permission.
[NYT] David Brooks: The Waning of I.Q.
I.Q. is like a black box. It measures something, but
it's not clear what it is or whether it's good at predicting how people will
do in life.
http://select.nytimes.com/2007/09/14/opinion/14brooks.html
J
====
Chris Brand here:
It is notable that "op-ed columnist" David
Brooks (while doubtless shocking some lefties by admitting 48% heritability
for IQ) does not quantify for NYTwits any of the non-IQ factors that he
fancies to be so important in life. He shows no sign of having read The Bell
Curve, Race, Evolution & Behaviour, The g Factor, or IQ & the Wealth of
Nations. Brooks thus fulfils the prediction in Heredity:
"Despite the limitations of their statistical
databases and methods, Lynn and Vanhanen have launched a powerful challenge
to economic historians and development economists, who prefer not to use IQ
as an analytical input. It is likely therefore that this work will be
studiously ignored, whereas it urgently needs to be refined and built upon."
Henry PALAIRET (economic historian), 2004.
Heredity 92, 4, 61-3. Reviewing R. Lynn & T. Vanhanen, IQ and the Wealth of
Nations. Westport, CN : Praeger.
Nor should Brooks be allowed to get away with
implying that there are vast emotional/artistic riches and complexities that
fail to be captured by general intelligence:
"The chief determiner of human conduct is the
unitary mental process which we call intelligence.... ....The intelligence
controls the emotions and the emotions are controlled in proportion to the
degree of intelligence.... It follows that if there is little intelligence
the emotions will be uncontrolled and whether they be strong or weak will
result in actions that are unregulated , uncontrolled and, as experience
proves, usually undesirable. Therefore, when we measure the intelligence of
an individual and learn that he has so much less than normal as to come
within the group that we call feeble-minded, we have ascertained by far the
most important fact about him."
H.H.GODDARD, 1919, Psychology of the Normal and
Subnormal. New York : Dodd, Mean & Co.
"Making art is one of the manifestations of
human intelligence; it is difficult to establish its limits, reasoning and
needs. There can be no such thing as spatial painting or sculpture, just a
spatial concept of art. The element in space in all its dimensions is the
only possible evolution of spatial arhitecture. There exists an art which
cannot be for everyone, and this is also true of mankind's other creative
forms, humanity is checked by them and we owe our civilizations to this
process. The only freedom is intelligence." Lucio Fontana (Italian
'neo-Futurist' abstract artist artist and sculptor), 1953, speaking at
Galleria del Naviglio, Milan, 18 iv. Quoted in Matthew Gale and Renato,
Beyond Painting: Burri, Fontana, Manzoni. London : Tate Publishing, 2005.
"What was it that you loved about Dora*?" I
asked [art lover and critic,] John Richardson. "Her intelligence," he said.
"There was something magical about her."
* Dora Maar was the muse and lover of Picasso
from 1936 till 1943 as well as being a painter in her own right. Picasso
later told his subsequent mistress, Françoise Gilot, that he had wanted a
former mistress, Marie-Therèse Walter "because she was sweet and gentle and
did whatever I wanted her to" but Dora "because she was intelligent." Mary
Ann CAWS, 2000, Dora Marr - with and without Picasso. London : Thames &
Hudson.
"..the strongest regiment of all in the army of
darkness: stupidity." Philip Pullman (author of His Dark Materials), 2005
interviewed in Times Higher, 22 iv, pp. 18-19.
"I've got a really high IQ, I was a smart girl.
If you're not and you haven't had a good education, and you've got a poor
diet, you're at a really massive disadvantage." Tracey Emin (postmodern
artist, 42, and still resolving to learn to spell; famous for exhibiting her
unmade bed, complete with soiled underwear, at the Turner Prize exhibition
in London, 1999), 2005. Interviewed by Martin Gayford, Daily Telegraph
(Magazine), 8 x. In 2007 she was made an Academician of the Royal Academy of
Arts - one of only 19 artists to achieve that distinction.
Strangely, Brooks could have drawn support for
his IQ-bashing from Charles Murray's latest:
"I am among the most emphatic of those who
think that the importance of IQ in living a good life is vastly overrated."
Charles MURRAY, 2007, 'Intelligence in the classroom.' Wall Street Journal,
17 i. But evidently Brooks cannot even be troubled to read the Wall Street
Journal.
My answer there was:
Am I mistaken in saying that IQ is still the best
single predictor of "success" however you want to define that? That is, if
you take a group of people in almost any profession or job description, and
rank order them by their "success" in whatever it is they do (say by having
a peer group rank order them) the best single predictor of that rank
ordering is IQ.
That certainly was the case for a long time: is it
different now?
When we did the grade prediction program, tests of
"g" were the first item in a multivariate prediction equation. I don't
recall all the weights and variances, but it used I think 6 items, and was
pretty good until the courts made UW stop giving it because it predicted
lwer grades (on average) for blacks. (Race wasn't one of the factors,
needless to say.)
So far as I know that is still the situation: IQ or
other measure of "g" is still the best single predictor. Understand I did
not say it was a great predictor, or that it was definitive; what I said was
it is the best single predictor. When combined in a multivariate prediction
program it can be very useful.
===================
On energy issues
Jerry, Once again the question begs an answer from
those who know. Where is the energy to produce hydrogen fuel and power
electric cars going to come from when several slightly warm days bring the
power grid to its knees. We do not have the power to run our society today
if every tool and appliance was turned on and put to work. Call full power
usage, instant black out! They will not build the quantity of sensible power
plants our current needs demand, so what will they do when millions of
hydrogen fueled cars hit the road along with millions more electric cars,
shut down the plants that make them to provide the power to run them? --
James Early
Long Beach, CA
We have spent the money that might have gone to
energy independence; it was poured into the sands of Iraq. And we did not go
in and get the oil pumping, which would have paid for the imperialist
adventure. Incompetent empire is not useful.
Now we have a tar baby, and I do not know how to
escape. If we came home now and put the money saved into nuclear power
plants we would come out ahead; but we will not do that.
=========
Subject: Fred's view of smart
people
Dear Dr. Pournelle,
I read the article, referenced in the 13Sep2007
"View". Originally, Fred's ramblings seemed arbitrary and dangerous. Now,
his statements are recognized as being based on his experience, and he
believes them quite strongly.
I quote one statement, and the spelling is his, "A
doctorate in psychology is a sure sing of confusion." "Sing" may be a typo,
a Freudian slip, or what he intended. His essays assure me that calling him
mistaken or wrong is about as smart as stepping on an animal trap to see if
it works. I wish both of you to continue your web pages, and will worry only
when the two of you competely agree with each other or any third party for
an extended time.
regards,
William L. Jones
wljones [dot] pe [at] verizon [dot] net
I think Fred knows that I have a PhD in
psychology... He's still a friend. I don't always share his views. He knows
that.
=========
Read this one:
: On the last days of the Republic and the Fury of the
Legions
This is a subject matter on which I can speak from
personal experience, perhaps other readers have more and better examples but
I would like to provide some input.
I have lived most of my life in a small country in
South America (Uruguay) which is notable for being so european that most
foreigners are amazed when they first come in. For almost a hundred years,
ending in 1973 this was the most democratic country in all of Latin America,
no ifs, ands or buts.
A series of guerrilla actions started by radical left
wing organizations (and don't you think that the Democratic party even in
its most extreme form compares) created a period of increasing unrest from
the late sixties on. This led to police action (which did not work) and
escalated to the calling in of the army during 1972, in 6 months the army
routed the guerrillas, never mind how well they were organized, they lost.
However, this let the genie out of the box, the
generals (and even more so the colonels and LtCnls) had complained for years
about the road the republic had been taking, the lack of government
attention to the country's problems, corruption, etc. When the guerrilla was
defeated they were not even thanked, even though the quasi-civil war was
still going on there were calls for their return to quarters without so much
as a "thank you" note.
At long last in June 1973 there was a coup, congress
was dissolved and we entered into a 13 year dictatorship, which at first was
hugely popular among many people who also thought government was not doing
its job, I remember this distinctly because I heard people in the street
commenting about the whole thing and literally saying "it was high time
someone did something like this, disorder and unrest were everywhere!". We
did get order, or rather ORDNUNG.
Let me point out that over the years I met many of the
actors in this whole political-military process and a very large majority
was made up of people who were highly educated, bright, and with the best of
intentions. However, they did not have the proper set of reflexes required
to run a country that has not been occupied in the course of a war, it is an
extremely common occurrence that military men will see everything through
the pattern of their training, thus every situation becomes (in their eyes)
a military one. The sad truth is that they don't make good ministers or
presidents and their governments tend to be extremely rigid and hardly
capable of modifying their own behavior.
The rub lies in that no matter how well intentioned
and bright military men do not know how to run a country except in extremely
specific circumstances, a normal civilian population WILL resent military
style government, and they also will make fun of those who govern.
So if you think you have a bad government now, please
think again, military government will be worse, it will certainly be much
worse for a country like the US with its tradition of freedom of expression.
A military emperor, no matter how good his intentions will eventually revert
to form, he can't help it, because to a general obedience by his
subordinates is both logical and obvious. Civilians are unruly, and the can
be nasty....
BTW the guerrillas who lost in 1972 are today the
democratically elected government, they have proceeded to show themselves as
utterly unfit, except in those few areas where they haven't changed the
previous government's economic policy.
All the best
Ariel
When I was in graduate school, Uruguay was held up
by my liberal-left professor of international relations as an example of
what could be in South America. But that was long ago.
Military government (Plato and Aristotle called it
Timocracy) is generally not stable. Military organizations are competent at
breaking things and killing people. Some officers are highly competent in
other matters, but they generally don't have the tools to accomplish their
goals. Still, the Roman Empire lasted a long time and did some glorious
things.
At the moment we have incompetent empire run by
civilian party hacks.
========
Jerry Predicts the Future;
Smokes!
Re: The Tom Snyder YouTube interviews: It's amazing
how well your predictions about what would become the Internet came true.
You even address electronic books. Your practical imagination was amazing.
But that suit . . . yikes! Although it was the height
of style back then.
Technology marches on: Your TI calculator on the show
can probably be outdone by $20 calculators hanging in blister packs at the
grocery store. And they say we don't live in a science fiction world.
Pete Nofel, editor pnofel@nelsonpub.com MAN – Modern
Applications News
=========
Global warming 'is good and is not our fault'
http://www.telegraph.co.uk/news/main.jhtml?
xml=/news/2007/09/14/nclimate114.xml
"Mr Avery, a senior research fellow at the Hudson
Institute, an independent US think-tank, said: "Not all of these researchers
who doubt man-made climate change would describe themselves as global
warming sceptics but the evidence in their studies is there for all to see.
"Two thousand years of published human histories say
that the warm periods were good for people."
Shocking, isn't it?
Graves
Indeed
========
Re: Independence Gardens and
Knowing Nothing About Food Preservation
Last week reader Jim mentioned the loss of hands-on
knowledge for methods of food preservation in the general population. The
knowledge has not actually been lost with the attrition of the elder
generations, rather it is (like so much else) embedded in books. This is a
topic that can and should be addressed by one’s survival library. Rodale
Press has a great deal to say on the subject, but other publishers and
authors address the topic quite thoroughly as well. This is my own
collection, and all are available through Amazon (and I seem to recall that
clicking through to Amazon.com via the Chaos Manor website earns Dr.
Pournelle a bit of pocket change).
Root Cellaring - by Mike Bubel - ISBN 0878572775
http://www.amazon.com/Root-Cellaring-Simple-No-Processing-
Vegetables/dp/0878572775/ref=sr_11_1/102-5180122-6682552?
ie=UTF8&qid=1189799165&sr=11-1
Stocking Up: How to Preserve the Foods You Grow,
Naturally - Carol Stoner - ISBN 0878570705
http://www.amazon.com/Stocking-Up-Preserve-Foods
-Naturally/dp/0878570705/ref=sr_11_1/102-5180122-
6682552?ie=UTF8&qid=1189799314&sr=11-1
Putting Food By - Janet Greene - ISBN 0452268990
http://www.amazon.com/Putting-Food-Plume-
Janet-Greene/dp/0452268990/ref=sr_11_1/102-
5180122-6682552?ie=UTF8&qid=1189799555&sr=11-1
Food Drying At Home the Natural Way - Bee Beyer - ISBN
0312905467
http://www.amazon.com/Food-Drying-Home-Natural-Way/
dp/0312905467/ref=sr_11_1/102-5180122-6682552?
ie=UTF8&qid=1189799029&sr=11-1
And since you have so much free time on your hands,
Dr. Pournelle, I’m looking forward to your forthcoming Special Reports page
on survival libraries ; ).
Greg Hemsath
In my copious free time...
========
The Case of the Shrinking Kilogram
Jerry,
Right of out Erle Stanley Gardner..."The Case of the
Missing Mass."
<http://ap.google.com/article/
ALeqM5h7zc9eAtkfeQZl_nWVvesqlZKd_g>
"Shrinking Kilogram Bewilders Physicists By JAMEY
KEATEN
PARIS (AP) - A kilogram just isn't what it
used to be. The 118-year-old cylinder that is the international
prototype for the metric mass, kept tightly under lock and key outside
Paris, is mysteriously losing weight - if ever so slightly.
Physicist Richard Davis of the
International Bureau of Weights and Measures in Sevres, southwest of
Paris, says the reference kilo appears to have lost 50 micrograms
compared with the average of dozens of copies....."
Regards,
Charles Adams, Bellevue, NE
===================
TOP
CURRENT
VIEW Friday
|
