Distrusting research

A Johns Hopkins review of nearly 150 randomized controlled trials on children — all published in well-regarded medical journals — reveals that 40 to 60 percent of the studies either failed to take steps to minimize risk for bias or to at least properly describe those measures. ...

Overall, 41 percent of the 146 trials in the review had improper or poorly described randomization techniques. Industry-funded trials were six times more likely to have high risk for biased randomization than government-funded trials or those funded by nonprofit organizations. And past research, the investigators point out, has shown that industry-funded trials are four to five times more likely to recommend an experimental drug.

The researchers also found that most of the trials (57 percent) either failed to use proper techniques that ensure anonymity or "blinding" to the type of treatment a patient gets, or they failed to clearly describe these techniques. The technique, called allocation concealment, ensures that neither the researcher nor the patient can guess which treatment they will get. The method also helps ensure that the treatment of one subject will not reveal to either scientists or the patients clues about the treatment of the next subject. Trials involving behavioral therapies were four times more likely to have this problem.

Overall, nearly 20 percent of the trials used improper masking techniques to ensure that neither the patient nor the researchers know which treatment went to which patient.

A corollary to Sturgeon's law is that 90% of all people are bad at their jobs. Maybe it's 57% in this kind of research. But you'll notice the peer reviewers didn't catch it, either. These all got published in real journals. And some of it may not be bad research design but that "they failed to clearly describe these techniques" in the articles.

So I guess what ordinary consumers of popular science, which is what I am, should do is keep reasonable skepticism about reports of studies, waiting for confirmation before jumping to conclusions. Fat chance.