The New Yorker has an interesting book review, What Nate Silver Gets Wrong.

Can Nate Silver do no wrong? Between elections and baseball statistics, Silver has become America’s secular god of predictions. And now he has a best-seller, “The Signal and the Noise,” in which he discusses the challenges and science of prediction in a wide range of domains, covering politics, sports, earthquakes, epidemics, economics, and climate change. How does a predictor go about making accurate predictions? Why are certain types of predictions, like when earthquakes will hit, so difficult? For any lay reader wanting to know more about the statistics and the art of prediction, the book should be essential reading. Just about the only thing seriously wrong with the book lies in its core technical claim.

The review authors start to explain Bayesian statistics as follows:

A Bayesian approach is particularly useful when predicting outcome probabilities in cases where one has strong prior knowledge of a situation.

The review is interesting reading.  I could probably stand to read it again and pay careful attention to the examples.  Probability was never my strong suit.

The comments on the article also prove to be very interesting.  I chose to add my 2 cents worth in response to one comment.

“Unfortunately, the same guys got it all wrong in 2007 (The Financial Collapse)”

There is a much simpler explanation of why they got it wrong in 2007. The predictions about mortgage backed securities were based on historical data of traditional mortgages based on sound banking principles. When the demand for MBSs became too strong, the idea of mortgages based on sound banking principals was abandoned. The probability of failure for sound mortgages was used to predict the failure of unsound mortgages.

On top of that, the leverage was so high, that a small decline in real estate prices could wipe out the whole industry. In the historical data collected, leverage was never this high.

So the failure was to build a model based on a history that did not apply to the current situation and then think that model could cover the current situation.

I spent my career making software to simulate numerical models of integrated circuits.  Mine were not intended to be sophisticated in the statistical (probability) domain.  The sophistication, such as it was, was in the physics.  This experience has made me sensitive to trying to extend models into domains in which the model may no longer be appropriate.  As the technology in the semiconductor industry advanced over the 40 year lifetime of my career, effects that used to be negligible started to become dominant.  As a result, the physical effects include in the  models was constantly under revision.

January 30, 2013

I finally went back to look more deeply into the calculation that the woman getting a positive result on her breast cancer test had a 90% chance of the result being a false positive.

I put words into the equations for the factors that were being calculated to get a better understanding of the meaning of the factors in the equations.

( fraction of women who have breast cancer *
fraction of women who have cancer and get a positive result )
= number of women who get a positive result and actually have breast cancer
if all women who have breast cancer were tested.

( fraction of women who do not have breast cancer *
fraction of women who get a false positive )
= fraction of women who would get a false positive if all women who did
not have cancer were tested.

The flaw in the argument is that almost all women who have breast cancer get tested, and not all women who do not have breast cancer are tested. I bet a larger fraction of women who do have breast cancer get tested than those who don’t have breast cancer get tested. You don’t get tested unless there is some other indication that you might have breast cancer. The results of the calculation cannot be taken as correct until you get actual numbers to correct the flaw. The numbers may turn out to be right, but this explanation is no proof that they are.

Which goes back to my old maxim, if the results of a calculation are in wide variance from what your intuition tells you, then you had better double check to see which one is right. This is one reason why all good scientific programmers try to have an intuitive idea of what the result ought to be before they write a program to make the calculation. Either they do it by intuition (manual calculations) or some alternative mechanical way to make the calculation. How else can you test to see if you have made any mistakes in your program?