Across studies of human health and behaviour, mechanical prediction was on average about 10% more accurate than clinical prediction.
Why it matters when the plan changes
Most consequential decisions about people are still made by combining evidence in someone's head. William Grove and colleagues' 2000 meta-analysis compared the two methods on studies of human health and behaviour: mechanical techniques were on average about 10% more accurate, and clinical prediction was substantially better in only 6% to 16% of studies. The advantage held across tasks, types of judge and levels of experience.
The tension is that experts see things a formula cannot, and sometimes that matters. The same research found clinical predictions did relatively worse when predictors included interview data, which is where experts tend to feel most confident. The practical reading is to structure judgement rather than remove it: combine evidence consistently first, then let an expert adjust for information the rule does not contain, and record why. That record is also what shows whether the adjustments help.
In practice
A board compares two ways of shortlisting chief executive candidates. One relies on a panel's overall impression after interviews. The other scores each candidate on the same pre-agreed criteria and combines the scores by a fixed rule before the panel discusses them. The research favours the second as a starting point, with the panel adding what the criteria miss.
Evidence
Mechanical prediction techniques were on average about 10% more accurate than clinical predictions, and substantially outperformed them in 33%-47% of studies depending on the analysis.
William M. Grove, David H. Zald, Boyd S. Lebow, Beth E. Snitz and Chad Nelson, Clinical Versus Mechanical Prediction: A Meta-Analysis, Psychological Assessment (2000)Clinical prediction was substantially more accurate than mechanical prediction in only 6%-16% of studies.
William M. Grove, David H. Zald, Boyd S. Lebow, Beth E. Snitz and Chad Nelson, Clinical Versus Mechanical Prediction: A Meta-Analysis, Psychological Assessment (2000)Clinical predictions performed relatively less well when the predictors included clinical interview data.
William M. Grove, David H. Zald, Boyd S. Lebow, Beth E. Snitz and Chad Nelson, Clinical Versus Mechanical Prediction: A Meta-Analysis, Psychological Assessment (2000)People lose confidence in an algorithm faster than in a human forecaster after seeing both make the same mistake, across 5 studies.
Berkeley J. Dietvorst, Joseph P. Simmons and Cade Massey, Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err, Journal of Experimental Psychology: General (2015)
What it cannot tell you
The research compares methods for combining the same information and does not show that any particular formula is good. A mechanical rule is only as sound as its inputs and its fit to the population, and it cannot use information that falls outside the variables it was built on.
Questions
Clinical prediction combines information through an expert's judgement. Mechanical prediction combines the same information through a fixed rule, formula or statistical model. Both can use interviews, tests and records; the difference lies only in how the pieces are weighed and put together to reach a forecast about a person or case.
Modest but consistent. William Grove and colleagues' 2000 meta-analysis found mechanical techniques about 10% more accurate on average, substantially better in 33% to 47% of studies depending on the analysis, and clinical prediction substantially better in only 6% to 16% of the studies examined.
No. It means the combination step should be structured. Experts remain essential for choosing what to measure, spotting information a rule does not contain, and taking responsibility for the decision. What the research discourages is an unstructured overall impression as the main method of combining evidence.
In Grove and colleagues' analysis, clinical predictions performed relatively less well when predictors included clinical interview data. A plausible reading is that vivid, personal information carries more weight in judgement than its predictive value justifies, while a formula weighs it only as much as the data warrant.
Partly because people distrust formulas after seeing them err. Berkeley Dietvorst, Joseph Simmons and Cade Massey showed in 2015 that people lose confidence in an algorithm faster than in a human after seeing both make the same mistake, a pattern they called algorithm aversion.