One of the grand difficulties in debating any issue, especially sitting around a few beers with peers, is the use of studies people have read and numbers people have heard. It is infinitely grating to me when this happens. I just finished reading another evaluation of a recent study of proximal intercessory prayer, and I felt I needed to comment. I don’t want to comment on the study itself, as it has been commented in detail by Steven Novella and PZ Myers, and I’m sure I’ve read comments from others that I simply can’t remember right now. Instead, I’ll comment on the overall use of studies and numbers in argument, and it will be merely my own random thoughts, and not some detailed screed expressing any kind of exacting science. So don’t quote this in conversation sitting around a few beers with peers.

First off, I want to address the term “average”. It amazes me how often that term is absolutely abused in conversation. People will say things like, “Well, your average American thinks Obama’s doing a terrible job” without having any reference to what the average American is. People will say things like “On average, I’d say the economy is doing well” but have no mathematics to back that up. Average is a mathematical concept. In my statistics class, we learned that there were different mathematical formulas that all wound up under the heading of average. We focussed primarily on median, mode, and arithmetic mean, but there is also geometric mean and harmonic mean. All of these concepts when applied to a given data set will yield different results. The arithmetic mean (the formula we most commonly associate with average) is where you add up all the numbers and divide by the number of results. The median is the middle value of the data set when it is sorted. The mode is the most commonly occurring number in a data set. For example, using the data set 2, 9, 1, 9, 6:

Arithmetic mean: 5.4
Median: 6
Mode: 9

All three numbers can be considered the average of the list, but clearly that number is different depending on which we use. It is not at all uncommon for confirmation bias to ensure that we choose the particular average that best defends our position. All of these results can be misleading in the right context.

My statistics teacher was a very unique person. He taught us both stats and database administration, and yet he was also a poet who had won the Governor General’s Award. I found him engaging and humorous, and whenever anyone would use the word average in conversation with him (at least in stats class) he would counter loudly with “What’s the Standard Deviation?” He was adamant that we understand that even if we knew what the arithmetic mean of a data set was, that hardly meant we understood the true breadth of the data. Just because we know the average does not mean that we understand the variability of the data set.

So when we’re sitting around those beers and someone says that on average, people believe in God, that means nothing. To understand the data properly, there are many more questions that need to be asked. What average are we talking about? How did they come to this answer? Did the question ask if the person believed in God or was it simply left at a creator? How widely varied were the responses? How many choices were there, and how was the data assembled? And on and on and on.

And then there is the issue of people who quote studies. The sad truth is that most of the time when people talk about studies they’ve read, what they really mean to say is “studies I have seen the abstract for” or “studies other people have read and then provided an analysis of that I may or may not have read more than a headline for”. Most of the time, we’re taking the study’s word for it based on the abstract or someone else’s interpretation, and that is foolish at best.

As well, there is the question of how well put together the study was. All studies should have certain key factors in order for us to consider them truly reliable. There are best case scenarios for studies, such as the controlled double blind study, but there are certainly times when a double blind is simply not possible or unethical. You wouldn’t give cancer patients a placebo with no other preventative measures to use them as a control in studying the efficacy of a cancer treatment, that would mean condemning them to death. Also, for a study to be truly reliable it has to properly rule out the other variables, from confirmation bias to fluctuations in temperature to goodness knows what. And all studies should be repeatable. If a study is properly performed, then recreating the process in another location with other researchers who diligently follow the process should yield results that confirm the findings in the study.

For the most part, when we’re chatting with our pals about the studies we’ve read, we don’t have a clue how valid the findings are. They are nothing more than headlines we are spitting out. Take for example this recent study on acupuncture. If we trust the abstract and the spin on the study provided by the people who favor it’s results, then it proves that acupuncture is an effective treatment. However, digging deeper into the information shows that the claims of the researchers may not be quite what the data showed. This can probably be chalked up to exuberance, but the point is the same. This study makes claims that are not supported by the data. And yet, in a conversation I could easily say “I read a study by some researchers who proved that acupuncture was an effective treatment.”

Don’t get me wrong, I do this too. I try to understand the information I use in a conversation, and if I’m uncertain on the facts I try to ensure that I communicate that. But I’ll be the first to admit that sometimes I am too hasty in my absorbing the broad strokes. I think we all do it, but it’s something we all need to be a little wary of.

Jim