A Note on Scientific Integrity

I discussed this already in a recent article in Weightology Weekly, but I feel that it is necessary to repeat it here as a blog post.

As some of you already know, I published a study in the American Journal of Clinical Nutrition in 2006 (although the actual research was performed in 2004).  The conclusions of this study were that low-carbohydrate intakes provide a metabolic advantage over higher carbohydrate intakes for weight loss.

However, information I've learned over the past year, along with newer and better studies, indicate that my conclusions were wrong.  Here are my comments from the Weightology Weekly article:

I mentioned earlier about how I now consider my 2006 meta-analysis conclusions to be unreliable.  The paper was a good paper for my knowledge level and the available data at the time.  However, science is always changing, as newer and more reliable data is collected, and as the knowledge base grows.  Any scientist with integrity should be willing to change his mind about an idea, if new information indicates that the old information may have been wrong.  I have learned much since the publication of that paper, and my new knowledge now tells me that the paper’s conclusions were likely wrong.  Among the reasons why I now consider that paper unreliable:

• I treated protein and carbohydrate as independent variables in my analysis.  However, because low carbohydrate diets are usually high in protein, the two are not truly independent.  This confounds the results of my analysis; regression analysis assumes that your predictor variables are independent of each other.  My analysis violates this assumption.
• I incorporated many studies that involved self-report of food intake.  It is known that self-report of food intake is unreliable for determining calorie intakes in overweight people.  To try to reduce the effect of this problem, I only incorporated self-report studies that had an objective, biological marker that indicated diet compliance (such as urinary nitrogen excretion to estimate compliance with a high protein intake).  However, the problem with this is that none of these markers are objective markers of actual calorie intake.  Thus, calorie intake estimates could still be considerably off, even if there was evidence of compliance based on another marker.  This is a big problem if you’re trying to determine if there’s a metabolic advantage.  If a low-carbohydrate diet offers an additional advantage in regards to satiety, then there would be systematic underreporting among individuals on high carbohydrate diets.  This systematic underreporting would show up as a “metabolic advantage” in a meta-analysis.
• I looked at body composition in this paper; however, if you’ve read my series on body fat testing, you now know that fat-free mass hydration and changes in body water can introduce significant error into measurements.  Because low carbohydrate diets have strong effects on body water, this would introduce systematic error when comparing body fat measurements between a low carbohydrate and high carbohydrate diet.

These flaws in my analysis, coupled with the new data from the well-controlled studies I discussed in this article, indicate that my conclusions were wrong.

I am still very proud of this paper.  It was an enormous amount of work; I had to read through over 700 diet studies, and so my knowledge of the scientific literature on diet increased exponentially.  I learned a lot during the process, particularly about statistics and meta analysis.  In fact, if it weren't for this paper, I would've never been able to do my meta-analyses on resistance exercise volume and strength/muscle size gains.  One can be proud of a paper even if the paper is eventually shown to be wrong.  This is the beauty of science...always self-correcting as the body of knowledge grows.

If any of you ever find someone referencing my study as evidence of a metabolic advantage, please refer them to this blog post.