"Who says? Compared to what?"

Imagine trying to make a decision about what make of car to purchase. You see an advertisement for a particular make of car that claims, “Eighty-eight percent of those surveyed selected this particular make over a Mercedes Benz in a head to head selection.”

That sort of claim gets your attention. Pretty convincing, right?  How could you not take a very close look at the make of car that was nearly unanimously selected over the car considered to be the apex of quality?

You take a closer look at the car advertised to be more desired than a Benz.  It isn’t comfortable, gas mileage is subpar, and maintenance is expensive. How could this line of cars outperform Mercedes in a head to head competition?

When you look closer into how the actual survey was conducted you find that  the survey was designed and conducted by the car maker that was not Mercedes Benz.  You also discover participants in the survey were asked to choose between  a 25 year old Mercedes and brand new model from the other car maker.  In addition to that, you discover that the 88% number was inflated by providing additional information to survey participants who initially selected the 25 year old vehicle.  Now what do you think?  

By now you are probably thinking how outrageous the above example is.  Nothing like that ever happens in the real world.  

You are right it is outrageous.

 Your are wrong in that it never happens (in the real world).

What was described above happens ALL the time in world of education.  How can that be?  A company promoting a program claims to be superior to something else that is being used.  The claims have to be true because the claims are supported by research. But, just like above, what do you find when actually scrutinize the research that supposedly supports those claims?

More often than not you find that the person or persons conducting the research was associated with the company selling the product.  Are you surprised that the results of the research find the product being sold find it to be superior?

In research conducted comparing one product to another it is not unusual to compare the product being sold to some vaguely described alternative.  A frequent comparison is made between students, “. . .being taught using a ‘traditional method,’’ or, “ . . .to students being taught using a ‘lecture method.’”  

What is a traditional method?  What is a lecture method?  Were the teachers who were using these methods (whatever it is they mean) judged to be effective and above average teachers? Or were the comparison group teachers simply ineffective?  Most research does not provide descriptions.

When I read “research” that does not specifically describe the comparison group, it is impossible for me to know what the authors of these studies mean. But I, get a funny feeling their brother-in-law owns a car dealership that sells inexpensive cars that are overwhelming preferred over a Mercedes Benz.

In the field of education we are constantly being sold something new - or at least new again.  If on first examination, based on your classroom experience, it simply does not make sense you need to take a closer look at the research that supports the claims.

Please do NOT take on faith any publisher's claim of, “This program is researched based.”  Look at the research yourself. As you read ask yourself, “Who says?” and “Compared to what?”