As an academic podiatrist teaching at one of our colleges and running a residency program, I spend a reasonable amount of time participating in and thinking about journal clubs and the medical research. I typically participate in three to four journal clubs per month, reviewing somewhere around six to ten articles over the month. Some of my faculty partners hate journal clubs, and I don’t blame them. These things can sometimes get a little long in the tooth, especially if we delve deeply into the biostatistics part. Luckily for me, I have a couple of strong academic partners for this part.

Journal Clubs Uncover the Truth About Clinical Research Articles

I’ve always felt journal clubs are an important part of both the educational process for our students and residents, but also a key method to remain current and active in an ever-changing field field. They also serve as a key component of evidence-based practice. Given my feeling on this topic, I find it especially sad that at least 95% of the time, the articles we review turn out to be junk.

---

At least 95% of the time, the articles we review turn out to be junk.

---

It’s incredibly rare that our journal clubs come out at the end saying, “Wow, this article is really going to change how I practice podiatric medicine and surgery.” Sad, right?

---

Even the Journal of Bone and Joint Surgery has published research with poor methodology. Publishing an article with a poor evidence level is just putting lipstick on a pig. It may have pretty lips, but it’s still a pig.

---

Let me now disabuse you of the idea that this is just a podiatry-ortho problem. In April 2015, the Wellcome Trust had a symposium on just this subject in London that looked at the “reproducibility and reliability of biomedical research”.¹ They state that possibly as much as half of the scientific literature may be untrue.¹ Is there any evidence to this?

---

Possibly as much as half of the scientific literature may be untrue¹

---

And Ends with the Citations

If the abstract is the beginning, what about the end? How accurate are the citations? Luo, et al analyzed the citations and quotations in 249 references and 408 quotations from 25 articles published in five orthopedic journals³. They found a citation error rate of 41% (103 errors out of 249 references) and quotation error rate of 20% (80 errors out of 408 quotes). Not even the references are accurate!

And the Study Design Isn’t so Great Either

How about the middle? If the beginning and the end contain inaccurate information, what about the actual research? For the sake of time (and my own sanity), let’s take one representative part of the methods of research studies: the statistics. Parsons and colleagues surveyed 100 orthopedic journal articles using a validated survey model.⁴ They found significant failing of study design, statistical analysis, and the presentation of results:

	In 17% of studies the conclusions were not justified by the results.
	In 39% of studies the researchers used the wrong statistical methods.
	In 17% a different analysis could have made a difference in the overall conclusions.

It’s the Best We’ve Got - Make it Better

Are you surprised by these results? I’m not. I can tell you that almost every journal article we review in our journal clubs have methodological or reporting errors. It makes one question the basis for our practice of medicine. I haven’t even brought up the concept of repeating studies. How many studies actually repeat a prior study to see if it was accurate? Very few. Some have suggested creating a research subspecialty of confirmatory research, which is a great idea.

So, what do we do with all this? We shouldn’t just give up and say, “That’s how it is.” Instead, we should all demand more from our researchers. We need to create clear methodological protocols for research to limit the amount of variability. If our journals publish peer-reviewed articles that are inaccurate, we need to write in to those journals and complain. Finally, until all of this inaccuracy changes, we each need to be highly skeptical of what we read. Don’t simply believe something because it is written. Use your mind and your own judgment to determine if a study has valid results and if you should use those results to change your clinical practice. Until that blue sky day of perfect research occurs (don’t hold your breaths), remain, as I, a skeptic.

---

Use your mind and your own judgment to determine if a study has valid results and if you should use those results to change your clinical practice

---