The scientific process: Disprove it!

26 Feb 2014 Posted by

I came across this quite excellent video earlier today.  It is simple, but effective, in conveying a key concept of scientific thinking, namely that we should seek to disprove, rather than prove our beliefs.  If you have five minutes, it’s well worth a watch, and I hope that it challenges you to think hard about your own ‘rules’.

It highlights how easily we can leap to false conclusions in our (human) desire to confirm what we already know.  As a consequence, our understanding of any subject, whether it is a rule for increasing numbers or a complex explanation for say, obesity, injury, or performance, is limited by how capable we are of recognizing what is not always obvious and accepting that our rule may not be the rule.

If you are and athlete or coach, for instance, think about how this change in thinking might affect your tactical planning and the methods you use to teach players skills, or to train.  How might you be blinding yourself to reality in your desire to confirm your belief?  Is the answer obvious, but you’re ‘choosing’ not to see it?  For scientists, well, it should be fairly obvious what the implication of the video is:


Don’t be the informercial.  Rather look for black swans

If you believe something to be true, you absolutely have to set out to disprove it, because otherwise you become the scientific (or coaching) equivalent of a television informercial (Verimark, for SA readers), which gathers the life-changing testimonies of many who have succeeded after trying Product X, and offers them as “conclusive evidence” or proof.  And we can start by seeking people for whom X does not work.

So look for the Black Swans, take pleasure in discovering a world that disagrees with you, because you recognize the incompleteness of your own rules, and that understanding grows more from disagreement than confirmation.  The extension of this realization should be to reign in your desire to dogmatically and definitively dismiss anything that does not fit your “rule” as fallacious and irrelevant.  The mere fact that these initially invisible rules exist should force us to reflect.


Wrecking balls, rules and reality

One very obvious example that is relevant to me right now is the 10,000 hour “rule” (I’m busy preparing a talk to give in Qatar soon).  There, you had this “rule” which was immediately exposed as unnecessary because so many expert performers were produced in less time.  These are the black swans of the training vs talent debate, and they force us to reassess our ideas around deliberate practice as a ‘primary determinant’ of expert performance (as per Ericsson, 2009).

Their discovery should not however be offered as proof that training is irrelevant, because theories do not have to replace one another.  Science is not a wrecking ball that swings wildly from one side to the next as it bashes down everything in its path.  However, the discovery of new observations and rules should drive a more complex understanding of just how expert performers are created.  Those who dismiss the observations of success in less than 10k hours are blinding themselves by a rule, and will never fully account for expertise.  Their common response, by the way, is to denigrate the quality of practice that fails to translate 10k hours into elite performance, which creates an unfalsifiable theory and is not even worth arguing against.  The well is poisoned in anticipation of rebuttal, and that’s not science.   The same kind of thinking happens for barefoot running and low-carbohydrate diets, incidentally.

In the course of my educational training, I was once shown a quote that said “Half of what we teach you is wrong. The problem is that we don’t know which half”.  Now, I don’t know who said it, or even if it is correctly quoted. But the point is that if even 20% of the “rules” that we are taught today are wrong, but we don’t know which 20% they are, then how can we be 100% certain about what we know today?  We are constantly learning new rules, and we must recognize that our initial thought, however obvious, is not necessarily true, because the same observation can be explained by more than one rule (as per the video).  When even the simplest explanation, a rule for ascending numbers, is potentially complex, how much more for the truly complex things in physiology, and sport?

So the challenge to us all, coaches and scientists, is to set out to disprove our belief, not to confirm it.  Test through ‘failure’, think differently about the things you’ve always assumed to be obvious, and maybe you’ll break through to a new level of understanding.


%d bloggers like this: