A Note on Scientific Integrity

I discussed this already in a recent article in Weightology Weekly, but I feel that it is necessary to repeat it here as a blog post.

As some of you already know, I published a study in the American Journal of Clinical Nutrition in 2006 (although the actual research was performed in 2004).  The conclusions of this study were that low-carbohydrate intakes provide a metabolic advantage over higher carbohydrate intakes for weight loss.

However, information I've learned over the past year, along with newer and better studies, indicate that my conclusions were wrong.  Here are my comments from the Weightology Weekly article:

I mentioned earlier about how I now consider my 2006 meta-analysis conclusions to be unreliable.  The paper was a good paper for my knowledge level and the available data at the time.  However, science is always changing, as newer and more reliable data is collected, and as the knowledge base grows.  Any scientist with integrity should be willing to change his mind about an idea, if new information indicates that the old information may have been wrong.  I have learned much since the publication of that paper, and my new knowledge now tells me that the paper’s conclusions were likely wrong.  Among the reasons why I now consider that paper unreliable:

  • I treated protein and carbohydrate as independent variables in my analysis.  However, because low carbohydrate diets are usually high in protein, the two are not truly independent.  This confounds the results of my analysis; regression analysis assumes that your predictor variables are independent of each other.  My analysis violates this assumption.
  • I incorporated many studies that involved self-report of food intake.  It is known that self-report of food intake is unreliable for determining calorie intakes in overweight people.  To try to reduce the effect of this problem, I only incorporated self-report studies that had an objective, biological marker that indicated diet compliance (such as urinary nitrogen excretion to estimate compliance with a high protein intake).  However, the problem with this is that none of these markers are objective markers of actual calorie intake.  Thus, calorie intake estimates could still be considerably off, even if there was evidence of compliance based on another marker.  This is a big problem if you’re trying to determine if there’s a metabolic advantage.  If a low-carbohydrate diet offers an additional advantage in regards to satiety, then there would be systematic underreporting among individuals on high carbohydrate diets.  This systematic underreporting would show up as a “metabolic advantage” in a meta-analysis.
  • I looked at body composition in this paper; however, if you’ve read my series on body fat testing, you now know that fat-free mass hydration and changes in body water can introduce significant error into measurements.  Because low carbohydrate diets have strong effects on body water, this would introduce systematic error when comparing body fat measurements between a low carbohydrate and high carbohydrate diet.

These flaws in my analysis, coupled with the new data from the well-controlled studies I discussed in this article, indicate that my conclusions were wrong.

I am still very proud of this paper.  It was an enormous amount of work; I had to read through over 700 diet studies, and so my knowledge of the scientific literature on diet increased exponentially.  I learned a lot during the process, particularly about statistics and meta analysis.  In fact, if it weren't for this paper, I would've never been able to do my meta-analyses on resistance exercise volume and strength/muscle size gains.  One can be proud of a paper even if the paper is eventually shown to be wrong.  This is the beauty of science...always self-correcting as the body of knowledge grows.

If any of you ever find someone referencing my study as evidence of a metabolic advantage, please refer them to this blog post.


Get the latest science on muscle gain and fat loss every month

Keeping up with the research is tough, so let me do the work for you. Consider signing up for the Weightology Research Review. I cover 8 studies per month and break everything down for you, so you don't need a PhD to interpret the data. You also get access to an archive of nearly 300 video and written reviews, evidence-based guides, Q&A's, and more. Click here to learn more.

Want some sample content before you buy?

Get Instant Access to Free Research Reviews!

 

9
Leave a Reply

avatar
4 Comment threads
3 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
5 Comment authors
ackersJames KriegerLloydLynMarie DayeCarbSane Recent comment authors
  Subscribe  
Notify of
ackers
Guest
ackers

So are you going to complete this mea culpe and submit a new re-analysis to the literature…or better yet…the same journal? I would argue that a blog post somewhere is meaningless to scientists who are going to read your article in the literature. There is no link from the lit to this.

Lloyd
Guest
Lloyd

Wow James. A lot of guts and humility from you to come out like this. I really wish that all science was as honest as can be no matter the thesis. I’m no low carb fan myself, but I feel very humbled by this read regardless. Thank you.

LynMarie Daye
Guest

Hey Carb Sane – I found the “Textbook of Biochemistry: Fourth Edition” by West / Todd / Mason / Van Bruggen published in 1966 at a flea market yesterday. I had to have it because I thought it would be a whoot to read too! A cursory glance shows that it includes a lot of theorizing as well as history of biochemistry. Now I just need to find some time!

CarbSane
Guest

James, this is the crux of my issue with not only LC “gurus”, but the mainstream as well. I remember in biology in college my text books were full of theories. Sometimes there were 3 or 4 competing theories discussed because we just didn’t know. I still have my bio texts from the 80’s here somewhere — it would probably be a whoot to read through them now sometime 😀 We are not always correct all of the time, and our beliefs change. Take politicians. One that doggedly sticks to some ideal even when the premise has been demonstrated wrong… Read more »