The Economist has an interesting article, Lessons of the 1930s: There could be trouble ahead.
In 2008 the world dodged a second Depression by avoiding the mistakes that led to the first. But there are further lessons to be learned for both Europe and America.
The article talks about opposing theories about the history and lessons learned. Unlike stories in many U.S. media outlets, this one does give some hints as to which theories have been debunked and which seem to stand up. The article is worth the read for the specific lessons learned.
That said, I am going to go off on a tangent about methodological lessons learned.
Reading this article had me thinking about a lesson I think I learned from the book Bad Science: Quacks, Hacks, and Big Pharma Flacks.
There is quite a difference between having a premise and seeking data to support it and having some data and trying to figure out what you can learn from it.
In the first case it is almost always possible to find data to support a theory if you look hard enough, massage it, and squint at it if you must, and ignore any data that casts doubt on your premise. In the second case you gather data in an unbiased way and see where it leads you.
I think you can see examples of both types of theories discussed in The Economist article.
If you think about the two situations more you see that it is very easy to fall into the trap of premise first and data next. You might start out in the unbiased way, and develop a theory. Then it is very easy to fall into the trap of just searching for more data to confirm your theory. This is called confirmation bias.
I can think of one way to search for more data while controlling the bias problem. When you set up your search or your experiment, always consider ways to disprove your theory as well as ways to prove it. Consciously think of data to look for or experiments to conduct that will disprove your theory. If the data you find or experiment you perform in an effort to disprove your theory fails to disprove it and your other searches and experiments tend to confirm it, then you are more likely to have found a theory that is closest to the truth.
Another problem arises when you get into a discussion (argument) with someone who disagrees with your theory. That is exactly when you are caught in the trap of trying to prove you are right. If the opposition presents some opposing data, your natural tendency is to look for flaws in that data. There is nothing wrong with that as long as you are also looking at the possibility that the opposition is right and their data do disprove your theory.
I write this blog entry as much as a reminder to myself as it is a lesson to anyone else.