The New Yorker has an interesting book review, What Nate Silver Gets Wrong.

Can Nate Silver do no wrong? Between elections and baseball statistics, Silver has become America’s secular god of predictions. And now he has a best-seller, “The Signal and the Noise,” in which he discusses the challenges and science of prediction in a wide range of domains, covering politics, sports, earthquakes, epidemics, economics, and climate change. How does a predictor go about making accurate predictions? Why are certain types of predictions, like when earthquakes will hit, so difficult? For any lay reader wanting to know more about the statistics and the art of prediction, the book should be essential reading. Just about the only thing seriously wrong with the book lies in its core technical claim.

The review authors start to explain Bayesian statistics as follows:

A Bayesian approach is particularly useful when predicting outcome probabilities in cases where one has strong prior knowledge of a situation.

The review is interesting reading.  I could probably stand to read it again and pay careful attention to the examples.  Probability was never my strong suit.

The comments on the article also prove to be very interesting.  I chose to add my 2 cents worth in response to one comment.

“Unfortunately, the same guys got it all wrong in 2007 (The Financial Collapse)”

There is a much simpler explanation of why they got it wrong in 2007. The predictions about mortgage backed securities were based on historical data of traditional mortgages based on sound banking principles. When the demand for MBSs became too strong, the idea of mortgages based on sound banking principals was abandoned. The probability of failure for sound mortgages was used to predict the failure of unsound mortgages.

On top of that, the leverage was so high, that a small decline in real estate prices could wipe out the whole industry. In the historical data collected, leverage was never this high.

So the failure was to build a model based on a history that did not apply to the current situation and then think that model could cover the current situation.

I spent my career making software to simulate numerical models of integrated circuits.  Mine were not intended to be sophisticated in the statistical (probability) domain.  The sophistication, such as it was, was in the physics.  This experience has made me sensitive to trying to extend models into domains in which the model may no longer be appropriate.  As the technology in the semiconductor industry advanced over the 40 year lifetime of my career, effects that used to be negligible started to become dominant.  As a result, the physical effects include in the  models was constantly under revision.

January 30, 2013

I finally went back to look more deeply into the calculation that the woman getting a positive result on her breast cancer test had a 90% chance of the result being a false positive.

I put words into the equations for the factors that were being calculated to get a better understanding of the meaning of the factors in the equations.

( fraction of women who have breast cancer *
fraction of women who have cancer and get a positive result )
= number of women who get a positive result and actually have breast cancer
if all women who have breast cancer were tested.

( fraction of women who do not have breast cancer *
fraction of women who get a false positive )
= fraction of women who would get a false positive if all women who did
not have cancer were tested.

The flaw in the argument is that almost all women who have breast cancer get tested, and not all women who do not have breast cancer are tested. I bet a larger fraction of women who do have breast cancer get tested than those who don’t have breast cancer get tested. You don’t get tested unless there is some other indication that you might have breast cancer. The results of the calculation cannot be taken as correct until you get actual numbers to correct the flaw. The numbers may turn out to be right, but this explanation is no proof that they are.

Which goes back to my old maxim, if the results of a calculation are in wide variance from what your intuition tells you, then you had better double check to see which one is right. This is one reason why all good scientific programmers try to have an intuitive idea of what the result ought to be before they write a program to make the calculation. Either they do it by intuition (manual calculations) or some alternative mechanical way to make the calculation. How else can you test to see if you have made any mistakes in your program?

Email I received from Friends of Bernie Sanders

January 24, 2013
U.S. Senator Bernie Sanders (I-VT)

This country faces major crises in terms of the economy and unemployment, the deficit, global warming, health care, campaign finance reform, education and a crumbling infrastructure – to name a few. In my view, none of these problems will be effectively addressed so long as one senator can demand 60 votes to pass legislation. The rule changes adopted today are a step forward in making the operations of the Senate more efficient and expeditious. They are not enough.

Most Americans grew up believing that in America the majority rules. That is not the case in the Senate. For many years now, especially since President Obama has been in office, it has taken 60 votes to pass any significant piece of legislation. When Lyndon Johnson was majority leader in the 1950s, he filed cloture to end a filibuster only once. Majority Leader Reid has filed cloture 390 times.

The Senate is not the House and the minority party must be treated with respect and given the opportunity to offer amendments and make their case in opposition. A minority must not, however, be allowed to permanently obstruct the wishes of the majority. That is not democracy. That is a perversion of democracy.

In my view, if a senator or a group of senators are strenuously opposed to legislation they have the right and duty to come to the floor and, for as long as they want, engage in a talking filibuster by explaining to the American people the reasons for their objection. They should not, however, continue to have the right to abuse arcane Senate rules to block a majority of senators from acting on behalf of the American people.

I created a petition to John Fischer, Mass Dept of Environmental Protection and Governor Deval Patrick which says:

“Do not lift the moratorium on incinerators in Massachusetts as long as there are far better alternatives to incineration.”

The people who run the SignOn petition web site have said, If you get 20 signatures on your petition, we’ll email it to at least 1,000 people in your area to help it grow (assuming of course that it’s consistent with MoveOn’s progressive values).

Will you sign this petition? Click here:

http://signon.org/sign/keep-the-incinerator – Pardon the misleading shortening of the URL.  SignOn truncated it just before the keyword.  It should have been  keep-the-incinerator-moratorium

After reading the story Barracuda Security Equipment Contains Hardcoded Backdoors, I immediately put these products on my s-t list.  My worry is how to remember that these products are on my list for as long as the problem exists.  I am afraid I might make a purchase after I have forgotten this story.

This is one reason to post this story on my blog and other pages that I have on the internet.  I hope this sticks in my mind.  Maybe yours, too.

Reader WayneP sent me a link to the item Nicholas Eberstadt: Yes, Mr. President, We Are a Nation of Takers.

A growing body of empirical evidence points to increasing dependency on state largess. The evidence documents as well a number of perverse and disturbing changes that this entitlement state is imposing on society.

I decided to do a little research on this item.  So far I have found The Big Lie About the “Entitlement State” .

Is the view that “entitlements”—government programs like Social Security, Medicare, and Medicaid—“will bankrupt the country” a “sensible conclusion”? No. It’s scare-mongering of the “OH MYGOD WE’RE ALL GOING TO DIE!” variety, completely unjustified by a sober look at data on government transfer payments between 1960 and 2010.

This second article is not a direct refutation of everything in the first article, far from it.  However it does delve a little deeper into the interpretation of raw numbers that might seem very scary if you leave out certain relevant context.

Obviously more research needs to be done to figure out which story is closer to reality, if in fact either story is anywhere near reality.

Do you ever wonder with the share of wealth and income migrating to the upper few percent in this country over the last 30 or more years, who are the real takers?  Well, at least who are the more successful takers?

How about the above for some undefined data taken out of context? Make of this what you will.

Here is an email I received from Elizabeth Warren on her view of the recent filibuster reform. I am glad to see that she realizes that this is but a first step in the process.

Steven, I’m disappointed by the filibuster reform deal we passed in the Senate last night. Since Election Day, hundreds of thousands of people all across the country have spoken out for big, bold rules changes to stop Senate gridlock. That’s not the bill we voted for in the Senate yesterday, and that’s why I’m really disappointed. But some change is better than no change at all. If my new colleagues can compromise to reduce the use of the filibuster, even just a little, then it’s a step forward. I’m new to the Senate, but I’m not naive. What happens if the new filibuster deal doesn’t work? What if the Republicans go straight back to the same playbook? Then we keep on fighting for more changes until we get it right. This vote shows that we need to remain vigilant, we need to work hard, and we need to stand strong for what we believe in. We got some change, and we’re not giving up. I want you to know that your participation made a difference. Without you — and thousands of other people — we’d have nothing and no hope for ending the gridlock. Now we have something, and we’re staying on high alert for what comes next. Thanks for being a part of this, Elizabeth

Daily Kos has the article Takeaways from the filibuster fight.  The author starts with the following:

Okay, before we decide once and for all whether this is a win or a loss, there are a few things I want out on the table. And the first point, appropriately enough, is that whenever you’re talking about the Senate, since the answer to any question about it is either “well, yes and no,” or, “it depends,” the answer to whether this is a win or a loss will be the same. Yes and no. And, it depends.

So, you know you are in for a long read before you find out.  To cut to the chase, I think the answer is that it was a fair start, but we have more work to do.

Rolling Stone has the article ‘Zero Dark Thirty’ Is Osama bin Laden’s Last Victory Over America by Matt Taibbi.

Now we have this movie out that seems to celebrate the use of torture against Arabs, and we’re nominating it for Oscars. Bigelow can say that “depiction is not endorsement,” but how does she think audiences will receive it in the Middle East? Are they going to sell lots of popcorn in Riyadh and Kabul during the waterboarding scenes?

Besides not wanting to see torture scenes, the bottom line, as Taibbi describes it above, is one reason we won’t be seeing this movie.

Perhaps I grew up on too many movies about World War II where we were taught to hate the Nazis because of their use of torture.  I can live with that.  I’d hate to think the next generations will be taught to love America because we use torture.  What is this world coming to?

RogerS lent me a copy of his DVD of Heist: Who Stole The American Dream?

The movie is best exsplained by its synopsis.

SYNOPSIS

HEIST: Who Stole the American Dream? is stunning audiences across the globe as it traces the worldwide economic collapse to a 1971 secret memo entitled Attack on American Free Enterprise System. Written over 40 years ago by the future Supreme Court Justice Lewis Powell, at the behest of the US Chamber of Commerce, the 6-page memo, a free-market utopian treatise, called for a money fueled big business makeover of government through corporate control of the media, academia, the pulpit, arts and sciences and destruction of organized labor and consumer protection groups.

But Powell’s real “end game” was business control of law and politics. HEIST’s step by step detail exposes the systemic implementation of Powell’s memo by BOTH U.S. political parties culminating in the deregulation of industry, outsourcing of jobs and regressive taxation. All of which led us to the global financial crisis of 2008 and the continued dismantling of the American middle class. Today, politics is the playground of the rich and powerful, with no thought given to the hopes and dreams of ordinary Americans. No other film goes as deeply as HEIST in explaining the greatest wealth transfer of our time. Moving beyond the white noise of today’s polarizing media, HEIST provides viewers with a clear, concise and fact- based explanation of how we got into this mess, and what we need to do to restore our representative democracy.

If you know how this country got in the mess we are in, this is a good refresher course. There may be some details you missed. In that case the movie fills in some of the gaps. If you think the cause of troubles is that Social Security and Medicare are too generous for us to be able to afford, what an eye opener this movie will be for you.

The movie is so distressing, I was almost going to joke that RogerS turns out to be no friend of mine.

The Massachusetts Chapter of The Sierra Club has an item about Backing the Ban on Incinerators.

Tell MassDEP Commissioner Kenneth Kimmell to maintain the ban on incineration and enforce waste bans. Comments will be accepted until February 15, 2013, 5pm, and sent to: John Fischer at dep.swm@state.ma.us or by mail to: John Fischer, MassDEP, One Winter Street, Boston, MA 02108. Please copy the Sierra Club on your communication at office@sierraclubmass.org.

As an example, here is the email that I wrote to the MASS DEP.

Subject:	Comments about the proposed lifting of the moratorium on incinerators
Date:	Mon, 21 Jan 2013 14:48:14 -0500
From:	Steven Greenberg <steve@ssgreenberg.name>
To:	dep.swm@state.ma.us
CC:	office@sierraclubmass.org

John Fischer,

Do not lift the moratorium on incinerators in Massachusetts.

In the Tellus Report, commissioned by the Commonwealth, there is all the necessary information on why the moratorium should not be lifted. Not only did Massachusetts commission this report, but the link takes you to the report posted on the Massachusetts web site.

The department that commissioned this report owes us all an explanation of why they would take action that is contrary to the technical, environmental, and economic information that they already have at their disposal.

Until such explanation is forthcoming, the moratorium should remain or perhaps be turned into a permanent ban.

/Steven Greenberg