A H.I.T.-less At-Bat, Part 5: My Response to Carpinelli’s Review of My Strength Meta-Analysis

This post represents part 5 of a multi-part series where I respond to Dr. Ralph Carpinelli’s critique of my meta analysis on single versus multiple sets for increasing muscular strengthIn part 1, I gave you some background on Dr. Carpinelli and my meta-analysis.  In part 2, I showed you how Carpinelli misleads the reader by omitting important information about my paper.  In part 3 and part 4, I showed you how Carpinelli's objections to my inclusion of the Rhea and Kemmler papers were not justified.  In this section, I show you how Dr. Carpinelli’s objections to my inclusion of a paper by Kraemer are not justified.

Kancel Kraemer?  Not Kw-ite...

Carpinelli lists 8 specific objections to my inclusion of the Kraemer paper.  Upon closer examination, only one of these objections is justified, but it still would not have affected the outcome of my paper.

  • Resurrected data from at least 15 years prior to publication.  Carpinelli gives no rationale as to why old data should be excluded from a meta-analysis, and does not give any criteria as to what the maximum age of data should be to be included.  The fact is, there is no good reason to exclude a study from an analysis simply because of the age of the data.
  • No control group, no control for repetition duration during 1 RM testing or training, and no indication that the trainers or those who assessed the 1 RM were blinded to the different training protocols.  Carpinelli listed these same objections to the Rhea and Kemmler papers; my responses can be found in my blog posts here and here.
  • Unprecedented 3-7 times difference in strength gains between groups in strong, previously trained (~2 years) Division I football players.  Carpinelli gives no rationale as to what the maximum difference in strength gains between groups should be to be eligible for inclusion in a meta analysis.  In fact, exclusion of a study simply because he feels the difference is too large could be considered data manipulation that would bias the results.  Carpinelli also provides no evidence that the differences in gains are "unprecedented."  The differences, while large, are not completely out of the ordinary.  For example, Marshall et al. recently reported gains that were 2 times larger in an 8-set group compared to a 1-set group in squats.  This was in trained subjects after only 6 weeks; the study by Kraemer lasted 4 weeks longer which would magnify differences even more.  In fact, in the Marshall paper, the 1-set group showed evidence of a plateau in the last 3 weeks of training, while the 8 set group continued to show an upward trend.  It should also be noted that I addressed outlier studies in my paper using a sensitivity analysis, where I removed studies one at a time to determine how much they impacted my results.  Removal of the study by Kraemer had no impact on my outcomes, something Dr. Carpinelli does not mention in his review.
  • Unsubstantiated speculation that the difference in strength gains may have been caused by greater hormonal responses, which were not measured.  This is where Dr. Carpinelli's objections to the inclusion of this study delve into the realm of absurdity.  It is not valid to exclude a study based on the study author's speculations or interpretations of his own data.  In fact, it is quite normal for a researcher, in the discussion section of the paper, to try to explain his results based on observations from other research.  In this case, Dr. Kraemer was attempting to explain his results based on previous hormonal data reported by Gotshalk et al., where greater hormone responses were found with a 3-set protocol as compared to a 1-set protocol.  Perhaps Dr. Carpinelli's lack of experience in publishing original data would explain his odd objection to an author trying to explain his data in a discussion section.
  • The author's claim that the answers to important training questions were determined in his role as a coach before he analyzed the data revealed a strong potential bias for a specific outcome.  Speculation as to whether an author may be biased is not a valid reason to exclude a paper from a meta-analysis.  In fact, is can be an author's biases that lead to the formulation of specific hypotheses.  All scientific researchers can be considered biased in a sense...biased towards what the data suggests to them.  A scientist makes observations, and thinks that the observations tend towards a certain direction...he becomes "biased" because his observations are leading him towards that bias.  He then formulates a hypothesis around that bias and tests it.  Obviously, an honest scientist will change his bias if his hypothesis is ultimately rejected as data is accumulated.  For example, I reported on this blog how I changed my mind on the "metabolic advantage" of low carb diets, and no longer agree with the results of my own 2006 scientific publication due to newer, better controlled data.  Regardless, it is simply inappropriate to reject a paper from a meta-analysis because of a personal suspicion of bias of the original author.  Dr. Carpinelli's belief of bias in the Kraemer paper is much more subjective rather than objective.  In fact, one could argue that Dr. Carpinelli has demonstrated extreme bias in his review of my paper, based on his omissions of important facts, misinterpretations of my data, and errors in his evaluation.
  • Forced repetitions in one group but not in the other group.  Of all of Carpinelli's objections, this is the one objection that could be considered a valid objection, as the two training protocols were not 100% identical between groups.  In the Kraemer paper, the single set group had forced repetitions at the end of a set, while the multiple set group did not.  However, this difference would not have impacted the differences between the groups; it has been shown in other research that there is no difference in strength gains between a group that does forced repetitions and one that does not.  In fact, based on "intensity of effort" as Carpinelli defines it, one might argue that the fact that the single set group did forced reps would have decreased the difference between single and multiple set groups, not increased it.  In other words, the forced repetitions, if they had biased the results, would have biased the results in favor of single sets, if intensity of effort is of utmost importance as Carpinelli believes.  Also, in the sensitivity analysis, I re-analyzed the data without the Kraemer paper, and my results were the same.  Thus, the inclusion of the Kraemer paper had no impact on my outcomes, something Carpinelli does not tell the reader.

Carpinelli also makes an error when discussing my reporting on the Kraemer paper.  He says that I "noted that experiment #2 was an unsupervised program."  I did not note that experiment #2 was unsupervised; I said that it was unspecified whether the program was supervised or not.  In my analysis, I categorized studies into either "supervised" or "unspecified".  Dr. Carpinelli apparently got "unspecified" and "unsupervised" confused.  Carpinelli then goes on to describe an anonymous questionnaire that Kraemer gave to 115 football players regarding their compliance to single set protocols (experiment #5).  The response was that 89% of the players reported using additional multiple set programs at home or at health clubs because they wanted to supplement their single set protocol.  Dr. Carpinelli then states:

If this were true for any of the trainees in Kraemer's experiment #2, it makes the reported differences between the single set and multiple set groups even more questionable.

However, if the subjects in the single set group in experiment #2 were truly sneaking out and doing multiple sets, this would have biased the results towards the single set group and decreased the difference between the groups.  Again, unsubtantiated speculation is not a valid reason to exclude a paper from a meta analysis.

Tossing Carpinelli a Bone

Even though nearly all of Dr. Carpinelli's objections to the inclusions of the Rhea, Kemmler, and Kraemer papers are not valid, let's assume for a moment that they are.  What would the results of my analysis be with the exclusion of all 3 of these papers?  Given that I still have my data tables, I can re-rerun the analysis without those papers.  When I redo the statistical analysis with those papers excluded, the effect size difference between single and multiple set groups only slightly decreases from 0.26 to 0.21, and is still statistically significant.  The mean effect size for single sets slightly increases from 0.54 to 0.57.  The mean effect size for mutiple sets slightly decreases from 0.80 to 0.79.  When I look at the effect sizes for the dose-response relationship, 1 set gets an effect size of 0.57, 2-3 sets gets an effect size of 0.76 (only sligthly down from 0.79), and 4-6 sets gets an effect size of 0.98 (up from 0.89).  In other words, my results are practically the same even if you remove all 3 papers that Dr. Carpinelli objects to.

In summary, nearly all of Dr. Carpinelli's objections to my inclusions of the papers by Rhea, Kemmler, and Kraemer do not hold up upon further scrutiny.  Even if you do remove all 3 papers, my results are still the same.

Click here to read Part 6, where I address Dr. Carpinelli's discussion of my 1-RM inclusive criterion.

 

4 Responses to “A H.I.T.-less At-Bat, Part 5: My Response to Carpinelli’s Review of My Strength Meta-Analysis

  • Jim Windus
    7 years ago

    Dear Jim,
    Sorry for not replying sooner. I am writing this with the understanding that I am NOT a scientist. I am a licensed psychologist and have been a personal trainer since 1979, and I have seen the industry grow, change, and morph in many more ways than most. That being said, I possess not a smidgen of the scientific knowledge you, or Dr Carpinellii possess.n In the field, I would consider myself a practitioner, one who interprets data and disseminates it for the public. The science that you and Dr. Carpinelli work very hard to put forth into a field filled with a wide array of ethics, morals, misconceptions, and out and out manipulation for money power and prestige.
    Now, that being said, I in good spirit invite you to re read some of your comments that you’ve written about Carpinelli. As I said above, in no way can I rebut your critique of Carpinellis article. That should be left to scientists like you, Dr. Kraemer, Dr. Winnett, Dr. Rhea etc. Your comments Jim, venture out of the realm of science and into the realm of philosophical and cultural agreement, where the validity claim lies not in data, but if there is substantial agreement upon most of the parties. Here is what you wrote:

    The same can be considered true of any letter to the editor in a journal, or a “critical review” like Carpinelli’s. Here are some more statements by some of my colleagues regarding Carpinelli’s paper…these were responses given to me well before I started to draft my own response:

    “his antagonistic, snide tone was really distracting. It was tough to get through that paper.”

    “he just sounded ornery, vengeful, & pissed off.”
    “You know, there are two impressions I got from reading this: 1. He’s got a personal ax to grind with you; 2. He doesn’t seem to understand m-a too well. Of course a m-a can be flawed. So can determining the mean if you neglect to include 4% of the data. Overall, I don’t consider this piece to serve academia or the scientific process.”

    Jim I wonder if you had the choice, would you write this in the same way?? What I mean is, by these comments do you truly feel you are representing the field of Exercise Science in a way that truly moves forth a healthy, respectful, open hearted dialogue that works to find meaningful solutions, or are you contributing to the degrading of colleagues due to disagreement?? Stuff like what your colleagues wrote, even if written in blogs, (which was my point about blogs..yes they have contributed much to free speech etc, but the first comment I read after after the Newtown shootings was from someone who wished he could have blown up both the mother of the shooter and the shooter. Hardly enlightening, don’t you think??) damages what is utilized by professionals like myself, who work with the public. Finding colleagues to “pile on” in a blog solves little in my mind, and only hurts what is put out to the public. In my opinion as a licensed psychologist, these comments much more reflect on the ones who said them than they do Dr. Carpinelli. This is truly a sad statement.
    Since you are writing on the level of agreement, I did some looking, and there is considerable disagreement with what your colleagues state about Dr. Carpinellli. The previous editor of the Journal of Exercise Physiology Online highly, and I might add, quite professionally, praised Carpinelli and his cohorts on their work several times, stating their work was “well written and researched in a very comprehensive manner” …and that they have contributed “outstanding work” that is a significant contribution to the scientific study of resistance training…and said editor agreed with their points about the peer review process needing a major overhaul, and that it was ripe with conflict and nepotism.
    Such dialogues Jim, if conducted in a healthy, meaningful, constructive manner, with the purpose of furthering the knowledge of science and exercise,can only help the lives of thousands. One should say, before writing, is what I’m writing furthering and honoring the field, or am I not being honest in my intent, and am I contributing to the debasement of science. You are ABSOLUTELY correct in that Carpinelli should not get a pass, no one should. But let it take place in a place where professionals like myself can benefit, and then can in turn use the information to in turn help others. Why not write to the editor the Journal of Exercise Physiology Online,and invite him to invite Dr. Capinelli to a point by point, online dialogue and debate, one that is conducted in a respectful, meaningful, and responsible way?? This is where a dialogue should take place, not in the blogosphere, which as I said, does have value, but one does need to learn how filter out the low level stuff that does not contribute much, if at all. I’ll end up with a quote by a certain scientist named Einstein, who said “”You cannot solve problems with the same level of consciousness that created them.” –Albert Einstein
    I hope you take this as a contribution Jim because that was truly how it was meant, Jim Windus

    “You cannot solve problems with the same level of consciousness that created them.” –Albert Einstein

    All of my responses are factual responses. They may sound like personal attacks, but instead they are harsh responses to Carpinelli’s points, including pointing out the hypocrisy of some of his statements.

    It should be noted that Carpinelli’s paper contained what could be considered personal attacks, questioning my objectivity and accusing me of confirmation bias. Even James Steele, an HIT advocate, noted as such in his blog post.

    Let me ask you…do you give Carpinelli a free pass because his personal attacks simply because his article was peer reviewed?

    This is partially what is wrong with internet responses. Anyone can write heavy handed semi scientific responses

    The same can be considered true of any letter to the editor in a journal, or a “critical review” like Carpinelli’s. Here are some more statements by some of my colleagues regarding Carpinelli’s paper…these were responses given to me well before I started to draft my own response:

    “his antagonistic, snide tone was really distracting. It was tough to get through that paper.”

    “he just sounded ornery, vengeful, & pissed off.”
    “You know, there are two impressions I got from reading this: 1. He’s got a personal ax to grind with you; 2. He doesn’t seem to understand m-a too well. Of course a m-a can be flawed. So can determining the mean if you neglect to include 4% of the data. Overall, I don’t consider this piece to serve academia or the scientific process.”

    Remember, these are comments about a paper that is published in a peer-reviewed journal.

    Blogging should be left to those who have a right to voice an opinion, but NOT have it pass as science. Thanks for the listen

    So are you saying that bloggers should not critically evaluate journal articles? Should bloggers not critically evaluate any area of science? For example, are you saying that the people over at sciencebasedmedicine.org should stop critiquing the poor science behind complementary and alternative medicine?

    Likewise, if a letter to the editor or “critical review” is published in a journal, does that automatically make it science?

    Carpinelli’s critique is nothing more than an opinion piece published in a journal. The fact that I’ve posted my response in a blog does not diminish the scientific accuracy or legitimacy of my response. If you feel that some of my responses are not legitimate, then please demonstrate where and how.

    (Quote)

    • What I mean is, by these comments do you truly feel you are representing the field of Exercise Science in a way that truly moves forth a healthy, respectful, open hearted dialogue that works to find meaningful solutions, or are you contributing to the degrading of colleagues due to disagreement??

      What I’m curious about is why do you not put forth the same statement to Dr. Carpinelli? Have you read Carpinelli’s review of my paper? Do you honestly believe that Carpinelli’s review “moves forth a healthy, respectful, open hearted dialogue that works to find meaningful solutions?”

      The reason I pointed out the comments of my colleagues is to illustrate that I am no the only one who has the opinion that Carpinelli’s review fell more into the realm of a biased personal attack rather than an honest attempt at a critical review.

      The purpose of my blog posts are not to solve a scientific problem. They are to respond to what is an inaccurate, dishonest, biased review of my paper, and to expose these characteristics of this paper. My response may be harsh, but when an individual portrays blatant hypocrisy in his review, then a harsh response is appropriate.

      Are you saying that dishonesty, bias, and inaccuracy should not be exposed?

      Stuff like what your colleagues wrote, even if written in blogs

      The stuff my colleagues wrote were not written in blogs. They were written to me in personal correspondence.

      In my opinion as a licensed psychologist, these comments much more reflect on the ones who said them than they do Dr. Carpinelli.

      How so? If someone reads a paper and gets the impression that the tone is inappropriate for a scientific paper, why would that negatively reflect upon the individual reading the paper? Are you saying that the tone for all scientific papers is always appropriate?

      Since you are writing on the level of agreement, I did some looking, and there is considerable disagreement with what your colleagues state about Dr. Carpinellli. The previous editor of the Journal of Exercise Physiology Online highly, and I might add, quite professionally, praised Carpinelli and his cohorts on their work several times, stating their work was “well written and researched in a very comprehensive manner”

      The statements by me and my colleagues have nothing to do with Carpinelli’s past work. It has everything to do with this one particular piece of work. Carpinelli’s past work may or may not be well researched and professional, but that is irrelevant to whether this particular piece is well researched and professional.

      and said editor agreed with their points about the peer review process needing a major overhaul, and that it was ripe with conflict and nepotism.

      I agree, but that is irrelevant to this particular piece of work by Carpinelli.

      am I contributing to the debasement of science

      How is pointing out inaccuracies, unprofessionalism, hypocrisy, and dishonesty in a particular paper “contributing to the debasement of science?” Should such characteristics go ignored when present in a paper published in a journal?

      Why not write to the editor the Journal of Exercise Physiology Online

      I am already constructing a paper for this particular journal.

  • Steven Goumas
    6 years ago

    That was fantastic, and damning. It seems that Jim would be your typical “plant”, I wouldn’t worry about it too much. Thank you for that great critique, I look forward to reading more by you. I’m still reading through “Body by Science” although I find the pitch very similar to Carpinelli and a lot of the scientific “proof” weak in a lot of places. Glad I found this critique.

  • Jim Windus
    6 years ago

    Dear Jim, as i said before i admire your knowledge base. It is truly beyond my ability to comment on, as is Dr Carpinelli’s. I truly feel the only way to further this conversation, which between the single /multiple set people is beginning to resemble the Israelis and the Palestinian conflict, is to have an editor or a journal ( say, Journal of Exercise Phiosiology Online) print letters from the two groups in alternating months. This way you, and say Dr Kraemer, can present your points , then a Carpinelli or someone else from the single set camp can present, and rebut. And it can hopefully be done in a respectful, collaborative way, where a mutually rewarding dialogue leads toward some resolution on this subject. practitioners such as myself only have much to gain. I wish you well should you decide to take on such an endeavor, Best to you Jim Windus

Leave a Reply Text

Your email address will not be published. Required fields are marked *