In the 1930’s, in the midst of a world wide economic depression, windstorms swept across the American Midwest and the two forces in tandem caused thousands of farm families to lose their homes. Many of them packed everything they could into farm trucks and migrated west to California where there were jobs, almost slavery in honesty, to be had in the rich farmlands. Although they came from all over, they were commonly referred to in the west as Okies, a nickname that has been common to the people from Oklahoma since the beginning of the 20th century.
It is said that the famous Oklahoma cowboy comedian, Will Rogers, commented on this migration by saying, “When the Okies packed up and moved to California, they raised the average IQ of both places.” Now the occurrence of just such an effect in statistics is a concern, and it has often been labeled the Will Rogers effect.
So how does that happen, here is a very simple example using easy numbers. Suppose a school has nine students, whose scores on a state exam last year were 60, 70, 80, 90, 100, 110, 120, 130, and 140. The school has identified students with scores below 100 as “under performing.” They report last year’s results with an average of 75 for the underperforming group, and 120 for the regular students. This year, they are introducing a new teaching approach for both groups, but along the way, someone decides that students scoring 100 will be moved to the underperforming group. This year the test results are 60, 70, 80, 90, 100, 110, 120, 130, and 140. The numbers look familiar, right? But now the school reports the average for the five underperforming students as 80 points, and for the regular students as 125.
Wow, Obviously the new strategy works, both groups improved by five points! But we know the truth the averages don’t tell, the scores are identical. Moving Joe average with the 100 point score from one group to another managed to raise the score of both groups, as Will Rogers had suggested.
It really does happen in real life, and statisticians have to be on the look out. In medicine for instance, we discover new methods of detecting illnesses, such as cancers, earlier and earlier. Now imagine that we have two groups, an “at-risk” group because they have been identified with a disease or disorder, and the “healthy” group. Usually they will have a lower life expectancy than the normal, not at-risk group. Now introduce a new identification measure into the mix. We begin to identify more people who have milder forms of the illness, and move them to the “at-risk” group. They probably have a shorter life span than the average of the “healthy” group, but they probably also have a longer life span than the “at-risk” population. This “stage migration” as it is called, will lead to an improvement in the life expectancy of BOTH groups, even without any treatment improvements. The person researching a new intervention will compare the hospitals results to those maybe five years ago (before the stage migration) and conclude that the new treatment is effective… NOPE, just the Will Roger’s effect.
Here is a footnote to the story about the Dust Bowl migration. The movie “Grapes of Wrath” was a harrowing depiction of the treatment of the Okies, and in general of Man’s inhumanity to man. I was told by a history professor in my youth that the Russian government thought this would be a great opportunity to expose to their people the impending collapse of capitalism by showing the movie in Russia. They offered free screenings all over the country, and millions of Russians saw it. And what was the result? It seems that they were desperate to get to America. Instead of the lessons their leaders had hoped they would see, most of them seemed to pick up on two facts. The poorest people in the US owned cars (almost no Russian below party level had a car) and they were free to pack up and move across country if they wanted. The law of unexpected consequences strikes again. I really don’t know how factual that story is (Gasp!!! Teachers may lie???) but it is a great story, true or false.
No comments:
Post a Comment