A H.I.T.-less At-Bat, Part 2: My Response to Carpinelli’s Review of My Strength Meta-Analysis

In Part 1 of this series, I introduced you to Dr. Ralph Carpinelli, a professor who has made a career out of writing critical analyses of papers in the field of weight training.  Dr. Carpinelli recently wrote a critique of my meta analysis on single versus multiple sets for increasing muscular strength.  In this multi-part series, I expose the numerous flaws and misleading aspects of his critique.  This series will be technical and academic at times, so skim the main points if that type of writing bores you.  In this section, I show how Carpinelli misleads the reader by omitting important information.

Already Off On The Wrong Foot

Carpinelli begins his critique with the following statement:

 The statistical process of a meta-analysis implies that theoretical and empirical science should be done by two different sets of people with different disciplinary abilities; that is, empirical research is performed by scientists and clinicians, but the interpretation of this research is performed by statisticians who decide what inferences should be drawn from the evidence [1].

This opening statement by Carpinelli is wrong in two ways.  First, it implies that only statisticians perform meta-analyses, which is not true.  In fact, while I am quite knowledgeable about statistics, I would not consider myself a statistician.  Many authors of various meta-analyses are not statisticians.  The second flaw is that Carpinelli has set up a false dichotomy; that meta-analysis somehow implies that the research is done by scientists and inferences are done by statisticians.  This is not true; all scientists draw inferences from their data, and statistics are only a formal tool to help draw those inferences.  The role of the statistician is to help select the most appropriate tool for helping to draw those inferences, based on the study design, data structure, and other important factors.

Carpinelli continues with the following statement:

The inclusion or exclusion of the studies in a meta-analysis is entirely based on the discrimination, opinions, and potential inherent bias of the statistician conducting the meta-analysis.

The H.I.T.-pocrisy of such a statement is apparent when you replace just a few words in the sentence:

The inclusion or exclusion of the studies in a narrative review is entirely based on the discrimination, opinions, and potential inherent bias of the author conducting the review.

In other words, Dr. Carpinelli's sentence could easily be applied to himself.  The major difference between reviews like that of Carpinelli, and a meta-analysis, is that the author of a meta-analysis employs a systematic, methodological approach towards gathering and analyzing data, so as an independent researcher could replicate the analysis if desired.  A narrative reviewer does not do these things.  As stated by Finckh and Tramèr:

Narrative reviews are prone to bias and error, while systematic reviews attempt to minimize bias by using a methodological approach.  In narrative reviews the methods for identifying relevant data are typically not stated, thus study selection can be arbitrary (influenced by reviewer bias).

Misleading The Reader

In the 2nd paragraph of his critical review, Carpinelli misleads the reader into believing that I would not be impartial in my meta-analysis.  He does so by quoting a sentence from a completely different narrative review that I published in 2010 in the Strength and Conditioning Journal.  The sentence read:

Thus, the number of sets can have a strong impact on the morphological and performance-based outcomes of a resistance training program.

Carpinelli then states:

Krieger did not cite any resistance training studies to support that statement, which may have been an early indication that readers were not going to get an impartial analysis of the topic.

This is a misleading statement by Carpinelli for the following reasons:

  • Carpinelli fails to tell the reader that this narrative review was published long after the publication of my meta-analysis.  Instead, he implies that I made this statement before I did my meta-analysis, which is not true.
  • Carpinelli fails to inform the reader that this narrative review was filled with references that supported that statement.  Carpinelli took this single sentence out of context.

Making Mountains Out of Anthills

In the next paragraph, Carpinelli takes issue with the fact that I interchanged the terms "meta-regression" and "meta-analysis."  He states that there are subtle differences between the two, and then comments how I never explained these differences.  It seems that Carpinelli is desperately trying to find anything to criticize, no matter how absurd.  A meta-regression is simply a type of meta-analysis, no different from how a regression is a type of statistical analysis.  Carpinelli fails to mention why an explanation of this is necessary, and how such an explanation would have had any impact on the outcomes of my paper.

Arbitrary?  No

Carpinelli then goes on to claim that my inclusion criteria were arbitrary.  This is not true.  There was a reason behind every set of criteria:

  • Must involve at least one major muscle group.   I want this analysis to be applicable to how the majority of people train.  Most people do not do exercises such as wrist curls or neck flexion.  Most people do exercises that target major muscle groups in some fashion, and even the American College of Sports Medicine recommends 8-10 exercises targeting major muscle groups.
  • Minimum duration of 4 weeks.  The duration of the study needed to be sufficient enough to not only observe strength gains, but also allow enough time for differences to arise between single and multiple set programs, if differences exist.
  • Single and multiple sets with other equivalent training variables.  The reason for this is obvious and is explained in the introduction and discussion portions of the paper, which Carpinelli fails to mention.  If you include studies where there are differences between the two groups other than the number of sets, you cannot be certain whether any observed differences are due to the different number of sets, or due to other differences in the training programs.  For example, there are resistance training studies out there which compare single-set, non-periodized programs to multiple-set, periodized programs.  Such studies were excluded because you cannot be certain whether the differences were due to the differences in sets, or the fact that one program was periodized and the other was not.
  • Pre-training and post-training 1-RM.  1-RM is by far the most common metric used to assess maximal dynamic strength, and is again applicable to how most people train.  Most people do not have access to variable resistance or isokinetic training equipment, which is why such studies that assessed strength using such equipment were excluded.
  • Healthy participants.  Orthopedic and musculoskeletal limitations could impact an individual's progress on a resistance training program, which is why studies that involved subjects with such limitations were excluded.
  • At least 19 years of age.  I wanted the study to results to be applicable to adults.
  • Sufficient data to calculate effect sizes.  This is obvious.  If there is not enough data to calculate an effect size, the study cannot be included in the analysis.

Carpinelli Misleads Again By Omitting Critical Information

Carpinelli goes on to make his most critical and glaring omission from his review.  He mentions how I used the sum of two 0-10 scale-based scores to rate the quality of the resistance training studies that I included in the analysis.  He fails to reference the two papers from which I obtained those objective scale-based scores; one paper was by Bågenhammar and Hannson, and the other was by Durall and colleagues.  By omitting this information, Carpinelli misleads the reader into believing that I arbitrarily scored the papers in my study.  He then goes on to offer arbitrary criticisms of three studies that happened to be among the highest quality scores in my paper, questioning how I awarded these studies the scores that they received.  If Dr. Carpinelli had read the references that I provided, he would have understood why these studies received the scores they did.  These two references laid out very specific criteria on how to score a resistance training study, and provided the scores for the studies.  Thus, with the exception of a few studies that were not included in these review papers, I did not score the studies; these studies were scored by the authors of these papers.  For example, here are the criteria set out by Bågenhammar and Hanson, with 1 point assigned to each criteria:

  • Subjects were randomly allocated to groups
  • Allocation was concealed
  • Groups were similar at baseline regarding the most important prognostic indicators
  • There was blinding of all subjects
  • There was blinding of all therapists who administered the therapy
  • There was blinding of all assessors who measured at least one key outcome
  • Measures of at least one key outcome were obtained from more than 85% of the subjects intially allocated to groups
  • All subjects for whom outcome measures were available received the treatment or control condition as allocated or, where this was not the case, data for at least one key outcome was analysed by "intention to treat"
  • The results of between-group statistical comparisons are reported for at least one key outcome
  • The study provides both point measures and measures of variability for at least one key outcome

Dr. Carpinelli misleads the reader to believe that I scored the studies, and questions my objectivity based on his own arbitrary analysis of three studies that I included.  However, the criteria are clearly laid out, and it is easy to determine how the studies received the scores that they did.  For example, studies by Rhea and Kraemer are two studies that Dr. Carpinelli takes issue with, and which Carpinelli cites as poor quality studies that should not have been included in my analysis.  Based on the objective criteria set out by Bågenhammar and Hannson, these two studies received a 5 out of 10.  It appears that Dr. Carpinelli advocates arbitrary decisions as to how to score the quality of resistance training studies, which is inappropriate when performing a systematic, quantitative review like a meta-analysis.

Carpinelli then goes on to discuss three papers that he thinks should have been excluded from my analysis, and summarizes his arbitrary reasons in a table at the end of the paper.  The papers in question are Rhea et al., Kemmler et al., and Kraemer.  In Part 3, I begin to look at Carpinelli's reasons why each paper should have been excluded; I will show you how his reasons range from invalid to absurd.

Click here to read Part 3.

15 Responses to “A H.I.T.-less At-Bat, Part 2: My Response to Carpinelli’s Review of My Strength Meta-Analysis

  • James Windus
    7 years ago

    Dear Mr. Krieger,
    I wonder why, if you feel so strongly that Mr. Carpinelli has misrepresented your data, research, and study, you don’t rebut him in a peer reviewed journal?? Some of your statements sound much more like personal attacks than healthy, point by point responses which should be at least part of scientific debate. This is partially what is wrong with internet responses. Anyone can write heavy handed semi scientific responses which can, and are go unchallenged scientifically. If you are so sure your response to Dr. Carpinelli’s would stand up to peer reviewed rigor, why not do so?? Healthy advances in science can only result. Blogging should be left to those who have a right to voice an opinion, but NOT have it pass as science. Thanks for the listen

    • Hi, James,

      I wonder why, if you feel so strongly that Mr. Carpinelli has misrepresented your data, research, and study, you don’t rebut him in a peer reviewed journal??

      As I stated in Part 1 of this series, I contacted the journal’s editor requesting that I publish a response and I got no reply. I am reqesting to write a more formal response in the Journal of Exercise Physiology Online, not only reviewing a few of Carpinelli’s critiques but one of Fisher as well.

      Some of your statements sound much more like personal attacks than healthy, point by point responses which should be at least part of scientific debate.

      All of my responses are factual responses. They may sound like personal attacks, but instead they are harsh responses to Carpinelli’s points, including pointing out the hypocrisy of some of his statements.

      It should be noted that Carpinelli’s paper contained what could be considered personal attacks, questioning my objectivity and accusing me of confirmation bias. Even James Steele, an HIT advocate, noted as such in his blog post.

      Let me ask you…do you give Carpinelli a free pass because his personal attacks simply because his article was peer reviewed?

      This is partially what is wrong with internet responses. Anyone can write heavy handed semi scientific responses

      The same can be considered true of any letter to the editor in a journal, or a “critical review” like Carpinelli’s. Here are some more statements by some of my colleagues regarding Carpinelli’s paper…these were responses given to me well before I started to draft my own response:

      “his antagonistic, snide tone was really distracting. It was tough to get through that paper.”

      “he just sounded ornery, vengeful, & pissed off.”

      “You know, there are two impressions I got from reading this: 1. He’s got a personal ax to grind with you; 2. He doesn’t seem to understand m-a too well. Of course a m-a can be flawed. So can determining the mean if you neglect to include 4% of the data. Overall, I don’t consider this piece to serve academia or the scientific process.”

      Remember, these are comments about a paper that is published in a peer-reviewed journal.

      Blogging should be left to those who have a right to voice an opinion, but NOT have it pass as science. Thanks for the listen

      So are you saying that bloggers should not critically evaluate journal articles? Should bloggers not critically evaluate any area of science? For example, are you saying that the people over at sciencebasedmedicine.org should stop critiquing the poor science behind complementary and alternative medicine?

      Likewise, if a letter to the editor or “critical review” is published in a journal, does that automatically make it science?

      Carpinelli’s critique is nothing more than an opinion piece published in a journal. The fact that I’ve posted my response in a blog does not diminish the scientific accuracy or legitimacy of my response. If you feel that some of my responses are not legitimate, then please demonstrate where and how.

      • James Windus
        7 years ago

        Dear Jim, I read your comments and thank you for the reply. I will have more of a chance to talk with you tomorrow, Saturday latest. Again thanks for your time, sincerely Jim Windus

      • Hi James,

        I thought I’d just leave a quick comment to say I have enjoyed most of your review and am pleased you have posted it here as in hindsight it has me reconsidering some of my present beliefs. I do still have some concerns about your meta-analysis but under further consideration I think you have conducted a very rigorous attempt at it and I wont spend time highlighting those specific concerns here.

        I just wanted to ask that, if you are interested to do so at some point in the future (PhD is taking most of my time now), I would be very interested to collaborate with yourself to address some of the issues with the present state of the literature to better address the question. I’ve myself begun to grow weary of some of the too and fro that’s going on and think instead conducting better research would better serve everyone’s interests, something which although it isn’t a valid criticism in and of itself of Carpinelli’s arguments, is something that his time would probably be better served in doing – and our own.

        Just let me know a yay or nay here, noting that it wont be until at least next academic year that I’ll find any time to do so, but if you are then I’ll drop you a more formal email at some point in the future.

        Thanks again and I look forward to the rest of your critique, and both myself and James Fisher also look forward to hearing your response to his JEP article.

        Take care,

        James

        • Hi, James,

          Thanks for your message. I’m glad you have enjoyed the review.

          I just wanted to ask that, if you are interested to do so at some point in the future (PhD is taking most of my time now), I would be very interested to collaborate with yourself to address some of the issues with the present state of the literature to better address the question. I’ve myself begun to grow weary of some of the too and fro that’s going on and think instead conducting better research would better serve everyone’s interests, something which although it isn’t a valid criticism in and of itself of Carpinelli’s arguments, is something that his time would probably be better served in doing – and our own.

          Yes, I would be happy to collaborate with you, time permitting. Drop me a line in the future when you have some time.

          James

  • Maverick wilson
    7 years ago

    Hello James, a few questions for a curious young student and trainer.
    can you speak to wether the groups involved in your meta-analyses were brought to full muscular failure during their sets? Or was there a predetermined number of repetitions to complete, based on how many reps the subject should be able to complete at X % load of the subjects 1RM?
    Can you adress your point of using the 1RM as a measurement for strength improvements? I don’t believe that stating that the 1RM is the most common method for exercise enthusiasts and athletes to measure strength makes it valid means of testing improvements in strength while conducting research when there are more accurate means of testing.
    Can you confirm or deny that exercises that demand higher degrees of coordination, such as the bench press, require more skill to preform than an isolated isometric exercise? And how increased neuromuscular coordination could be the dominant reason for the increased strength seen in this exercise, used for 1RM strength testing, rather than an increase in muscle CSA leading to an increase in strength?
    Additionally, why are the subjects being tested pre and post at a load that they were not training at? Would a direct measurement of force production via isokentic strength training machines not be a more accurate way to discover increases in a given muscles strength?
    also, regardless of the size of the muscle group being trained, is the stimulus and adaptive response process of the skeletal muscle group being trained not the same?

    • Hi, Maverick,

      can you speak to wether the groups involved in your meta-analyses were brought to full muscular failure during their sets?

      Yes, they were in all studies except one.

      I don’t believe that stating that the 1RM is the most common method for exercise enthusiasts and athletes to measure strength makes it valid means of testing improvements in strength while conducting research when there are more accurate means of testing.
      Can you confirm or deny that exercises that demand higher degrees of coordination, such as the bench press, require more skill to preform than an isolated isometric exercise?

      It is a common myth set forth by the H.I.T. camp that 1-RM testing is not an “accurate” means of testing maximal strength. They argue that strength increases are the result of “skill” improvement instead. However, they never provide any evidence for this. In fact, there is no evidence at all that a 1-RM test is somehow less “accurate” than an isometric test. The problem with isometric testing is that it is completely dependent on joint angle, and improvements in isometric strength may not translate into improvements in dynamic strength. Remember, it is dynamic strength that is of relevance to real life activities. In real life, your ability to lift something off the floor matters much more than your ability to isometrically produce some force in your knee joint at a 45 degree angle.

      And how increased neuromuscular coordination could be the dominant reason for the increased strength seen in this exercise, used for 1RM strength testing, rather than an increase in muscle CSA leading to an increase in strength?

      There is zero evidence that increased neuromuscular coordination is the dominant reason for increased strength in an exercise like a bench press. Are you going to tell me that, if I increase my bench press by 100 pounds over a few years, that’s all neuromuscular coordination? Are you saying that increases in bench press strength never correlate with increases in muscle size?

      The scientific data has shown a dramatic increase in muscle protein synthesis within hours of a single training session of an exercise like a bench press in people who have no training experience. Protein synthesis is the driving force behind increases in muscle CSA.

      Also, neural adaptations are an important component of strength training adaptations, and your nervous system is an integral part of your ability to produce force. You can’t separate the neural adaptation component from the structural adaptation component when it comes to increases in maximal force production. In fact, increases in isometric force production are just as much a product of neural adaptations as increases in dynamic 1-RM performance.

      Additionally, why are the subjects being tested pre and post at a load that they were not training at?

      If you do anything less than a 1-RM, then it is no longer an assessment of maximal force production for that particular exercise. For example, if you test someone on a 10 RM, you have introduced an endurance component.

      Would a direct measurement of force production via isokentic strength training machines not be a more accurate way to discover increases in a given muscles strength?

      How is isokinetic testing more accurate? More accurate in terms of what? Force production in real life is not isokinetic in nature.

      also, regardless of the size of the muscle group being trained, is the stimulus and adaptive response process of the skeletal muscle group being trained not the same?

      I’m not quite sure what you’re asking here. Are you asking if the stimulus and adaptive response between different sizes of muscle groups the same?

      • Maverick
        6 years ago

        Hey James, sorry for the delayed reply to the answers your provided me.
        I recently read an other critique of your meta-analysis, published in the journal of exercise physiology (online), which left me with even more questions.
        http://www.asep.org/asep/asep/JEPonlineDECEMBER2012_Fisher2.pdf

        In response to your reply:
        “It is a common myth set forth by the H.I.T. camp that 1-RM testing is not an “accurate” means of testing maximal strength. They argue that strength increases are the result of “skill” improvement instead. However, they never provide any evidence for this. In fact, there is no evidence at all that a 1-RM test is somehow less “accurate” than an isometric test. The problem with isometric testing is that it is completely dependent on joint angle, and improvements in isometric strength may not translate into improvements in dynamic strength. Remember, it is dynamic strength that is of relevance to real life activities. In real life, your ability to lift something off the floor matters much more than your ability to isometrically produce some force in your knee joint at a 45 degree angle”

        Dynamic strength? how are increases in muscle mass, as determined from isometric strength testing and xray absorbitometry not dynamic? and how are increases in bench press strength not completely dependent on the joint angles of that movement? If this were the case than increases in incline, decline, and Dumbbell presses, should increases linearly with increases in bench press strength. This is not the case. also, when in real life are you ever performing a bench press action (i.e. lying on your back and pressing a bar) ?

        ” There is zero evidence that increased neuromuscular coordination is the dominant reason for increased strength in an exercise like a bench press. Are you going to tell me that, if I increase my bench press by 100 pounds over a few years, that’s all neuromuscular coordination? Are you saying that increases in bench press strength never correlate with increases in muscle size?

        The scientific data has shown a dramatic increase in muscle protein synthesis within hours of a single training session of an exercise like a bench press in people who have no training experience. Protein synthesis is the driving force behind increases in muscle CSA.

        Also, neural adaptations are an important component of strength training adaptations, and your nervous system is an integral part of your ability to produce force. You can’t separate the neural adaptation component from the structural adaptation component when it comes to increases in maximal force production. In fact, increases in isometric force production are just as much a product of neural adaptations as increases in dynamic 1-RM performance.”

        Agreed, there is obvious structural change as a result of using the bench press. But does an exercise with greater degrees of freedom not require a higher level of coordination to perform? Would a focus on coordinating between different muscle groups (including those outside the target muscles [i.e. pecs, anterior shoulder, triceps]), as a result of the increased freedom of movement, not take away from some of the volitional effort that would be put in to inducing fatigue into the desired muscle groups? and is the degree of fatigue not the stimulus for muscle growth?

        “How is isokinetic testing more accurate? More accurate in terms of what? Force production in real life is not isokinetic in nature”

        here you make an argument again for “dynamic exercises” having more relevancy to real life. I do not understand how anything outside of increased muscle mass has any transference to real life. exercise performance is a combination of both muscle size and strength and neural efficiency (i.e. skill). for a skill to be transferable it must be exact in nature, not similar. programming two different but very similar motor programs only results in confusion and a negative transference. can you again explain how a bench press would be a better measure than isokinetic testing?

        “I’m not quite sure what you’re asking here. Are you asking if the stimulus and adaptive response between different sizes of muscle groups the same?”

        Yes I was. You were dismissing some of Carpinelli’s study’s as they were using smaller muscle groups, such as the forearm. How are exercises for a smaller muscle group any less relevant than more compound movements if the process of increasing muscular size and strength is the same?

        Also, I would be curious to hear your response to the study which I linked at the beginning of this post.

        • Dynamic strength? how are increases in muscle mass, as determined from isometric strength testing and xray absorbitometry not dynamic?

          Isometric strength testing is not a measure of muscle mass. It is a measure of force production at a specific joint angle.

          Regarding DEXA, it is also not a measure of muscle mass. It measures fat-free mass, which includes internal organs, body water, etc. They are not the same thing.

          Dynamic strength refers to the production of force over a dynamic range of motion, not at a static single joint angle.

          and how are increases in bench press strength not completely dependent on the joint angles of that movement? If this were the case than increases in incline, decline, and Dumbbell presses, should increases linearly with increases in bench press strength.

          You are misunderstanding my use of the term joint angle. I’m referring to the angle of the joint where torque is occurring. For example, an increase in isometric torque produced by the quadriceps with the knee joint at a 90 degree angle may not translate to an increase at a 70 degree angle. Dynamic force production involves force production through a variety of joint angles, from the beginning of the movement to the end of the movement.

          But does an exercise with greater degrees of freedom not require a higher level of coordination to perform?

          To a certain degree, yes, but there is no evidence that this would impair the process of increasing strength or hypertrophy.

          Would a focus on coordinating between different muscle groups (including those outside the target muscles [i.e. pecs, anterior shoulder, triceps]), as a result of the increased freedom of movement, not take away from some of the volitional effort that would be put in to inducing fatigue into the desired muscle groups? and is the degree of fatigue not the stimulus for muscle growth?

          This is another erroneous assumption of the HIT proponents. There is no evidence that the “degree of fatigue” is the stimulus for muscle growth. “Degree of fatigue” is also a very vague concept, a concept I would consider in the realm of pseudoscience. What is meant by degree of fatigue? Depeletion of phosphocreatine stores? Decrease in muscle pH? Impairment of neural drive?

          I do not understand how anything outside of increased muscle mass has any transference to real life. exercise performance is a combination of both muscle size and strength and neural efficiency (i.e. skill).

          You are demonstrating the whole problem with the thought process of the HIT camp. You are assuming neural efficency and skill are the same thing. Where is the evidence that they are the same? And what is meant by “neural efficiency” specifically? Synchronization of motor unit recruitment? Ability to recruit all motor units? Ability to alter firing rates of motor units to enhance force production? The use of vague terms like “neural efficiency”, and loosely interchanging this term with a very broad concept like skill, are trademarks of pseudoscience.

          Yes I was. You were dismissing some of Carpinelli’s study’s as they were using smaller muscle groups, such as the forearm. How are exercises for a smaller muscle group any less relevant than more compound movements if the process of increasing muscular size and strength is the same?

          Why do you assume the process for increasing muscular size and strength is the same? There is a wealth of scientific data that indicates that there are a variety of mechanisms for hypertrophy, both mechanical and metabolic. Also, training status and the amount of use a muscle gets can actually alter a muscle’s response to training, which has been demonstrated in scientific research. The training response of a small muscle group that may not be heavily used will be quite different from a larger muscle group that will be more frequenly used.

Trackbacks & Pings

Leave a Reply Text

Your email address will not be published. Required fields are marked *