Weekends with Womack

What do weight-loss studies REALLY show? Looking behind the scenes

Every so often, one of the big medical journals will publish a study by hard working medical researchers who look for answers to the question, “Which weight-loss programs are effective for long-term weight loss?” (SPOILER ALERT: the answer is “none of them”).

A couple of weeks ago, the Annals of Internal Medicine published a meta-analysis examining a bunch of studies that themselves tested various commercial weight-loss programs for effectiveness. By “effectiveness”, they meant whether and to what extent the participants in the studies (not the programs in general) registered weight loss 12 months after starting the program. Some of the programs evaluated were:

  • Weight Watchers
  • Jenny Craig
  • Nutrisystems
  • Atkins
  • Medifast
  • Optifast
  • Slimfast
  • And a few others

So, what did the study show? Well, it depends on where you read about the study, and what you can read from the study. Of course, readers of this blog have been well-served by many posts that show and explain scientific evidence for how unsuccessful diets are. And, if we look behind the scenes at what’s actually in this study, we can see just how limited the evidence really is.

This matters for a lot of reasons, but two big ones come to mind: 1) Almost all of these weight loss programs are expensive—at least $100/month, and some cost upwards of $600/month. Most people can’t afford that, and if their insurance covers it, that’s money not being spent on other (more effective) health promotion treatments. 2) Given the dismal success rates of all of the programs, they set people up to fail and feel bad about failing (while paying money for the privilege). So let’s now take a closer look.

If you read about the commercial weight-loss program study in the popular news media, you heard a variety of messages. Both the LA Times and Reuters reported the most digestible results, which were that Weight Watchers and Jenny Craig were more effective for long-term weight loss, with 3% and 5% average weight losses at the end of a year.

Time provided a more thorough and nuanced report, in which the positive results are listed, but they’re put in perspective by medical experts who say that 3-5% weight loss is not much, and the design of the studies limits their applicability to real life (they’re not kidding—more on this in a minute). However, ever-optimistic about the possibility of long-term substantial weight loss, they quote several experts who maintain that 3—5% is a good start, and that a small amount of weight loss can nonetheless translate into better health measures (like blood pressure, blood sugar and cholesterol).

The Associated Press story is more upbeat about the study’s results, and even throws in a personal story of a former employee of one of the researchers who is now on Weight Watchers and has lost 7 pounds in one month. We’re now all supposed to think: Wow! Maybe that could be me, too! Or: Oh, I tried that program and gained back all the weight I lost and then some; what’s wrong with me?

So what’s the real story here?   What are we supposed to think about commercial weight loss programs based on this study?

Reading through a medical journal article takes some time, patience, and background knowledge. I happen to benefit from the technical assistance and experience of my partner Dan, who’s an internist at a community health center. And I also do research on obesity and behavior change, so I’m used to plowing through this kind of information. Still, deciphering the careful understatement of medical authors takes some doing.

Here’s what I found in a close read of the article.

In order to evaluate a study for effectiveness, you need to take into the account the following factors:

Duration—any study that lasts less than 12 months is useless, as we know that short-term weight loss can happen on just about any diet (grapefruits, anyone?). The hard (which is to say, virtually impossible) part is maintaining it long-term.

Adherence—this refers to how many people actually followed the diet plan of the study. Any study with low adherence rates is not helpful. Why not? Well, maybe they didn’t keep up the plan because the diet was not doable (or too expensive—more on this in a bit).

Attrition—this refers to how many people dropped out of the study for whatever reason (e.g. they moved, they had bad side effects, they lost interest, they didn’t like the diet plan, etc.). This is relevant for reasons similar to adherence.

Bias features—this is a bit complicated, but studies need to select their participants, evaluate their progress, and make conclusions in scientifically scrupulous ways. Any study with a high risk-of-bias is suspect (more on this, too).

Cost—the authors, being sincere and medical and all, actually want to apply their findings in the form of clinical recommendations for primary care providers. This means that cost matters (I mentioned this already but it bears repeating), as many patients can’t afford or aren’t willing to pay large sums for these commercial programs. Also, insurance companies that might consider extending coverage for these plans take cost into account.

Adverse outcomes – this is medical-speak for “when bad stuff happens to people”. For instance, in one of the diet plans (Health Management Resources, a very-low-calorie diet plan that includes meal replacement shakes), 6% of the participants had gall bladder operations during the study (which is many times higher than the incidence of such operations in the general population in North America), and 56% of them reported constipation. Both of these are adverse outcomes (although one of significantly worse than the other).

So here’s the inside skinny on these features of their study:

Duration: Only Weight Watchers, Atkins and Jenny Craig were subjected to 12-month studies. The rest of the programs were used in 3—6 month studies, some of which reported 2—10% weight loss results, but without follow-up, they don’t mean anything (do I have to bring up the grapefruit diet again?). There are lots of ways to lose weight quickly, but so far no systematic ways to maintain weight loss in the long term.

Adherence: Many of the studies didn’t measure adherence—it is hard to measure, as you either have to sequester people in a medical setting (this happens for some short-term studies, but it’s expensive and hard to find participants) or ask people to fill out diet recall forms. These are known to have problems with accuracy. So this means that when a diet plan study has low success, we don’t really know how much of that effect is due to the diet and how much of the effect is due to people not following it.

And in addition, we don’t know WHY people didn’t follow it—this is a question I’m particularly interested in. Does the food not appeal to them? Does it require preparation that they don’t know how to do? Does it conflict with some of their regular eating habits? Do their friends and family not support eating in this way? We don’t eat in isolation—a big study on social networks and obesity )showed that our friends’ body weights have a strong influence on ours. There’s more to say here, but then again, I’ve got a weekly column now (yay!), so I’ll leave it here for now.

Attrition: Attrition is easier to measure– you just count up who is still in the study at the end. Right. But again, like in the adherence case, we want to know WHY people drop out. The same intriguing questions remain, but these studies aren’t designed to ask them. Why not? Beats me.

Bias features: The authors came up with a way to rate the bias of each study, based on the ways participants were selected, ways progress was evaluated, and ways conclusions were drawn. They gave each study one of three ratings: Low-risk-of-bias, Medium-risk-of-bias, or High-risk-of-bias. It turns out that of the 36 studies included, most of them were rated medium to high-risk. And almost all of the Weight Watchers and Jenny Craig studies used to create those positive headlines were medium to high-risk. For instance, one of the studies selected their participants from women who had had breast cancer. Surely this is not a representative sample of the population. Moving on…

Cost: Most of them are expensive, some of the very expensive. As I mentioned above, they simply may not be worth the money—either yours or your insurance company’s.

Adverse outcomes: I already mentioned some adverse outcomes for the HMR very-low calorie diet that was reported in the study. But many of the studies that the researchers looked at didn’t report adverse outcomes. So we don’t know much about them. Let me be clear in saying that I don’t suspect that there are unreported serious medical conditions that are resulting from these diets. But, since very few of the studies examined lasted longer than 3—6 months, and none of them lasted more than 12—18 months, we don’t know about the most relevant adverse effect: regain of weight lost, or the yo-yo diet effect.


Yo-yo dieting has lots of negative effects, both physical and psychological. This is well-known in the medical literature. However, in their recommendations for clinicians, the authors of this study fail to mention the potential downside of yo-yo dieting for their patients. At the very least this suggests that any serious medical studies on weight-loss programs need to follow their participants for longer periods. Of course, studies are very expensive, and body weight variation is complicated. It responds to an array of influences that range from environmental to economic to ethnic to psychosocial. So this is not easy science to do or understand.

One area of research I strongly support is trying to better understand people’s conceptions of health and healthy eating and healthy activity. If public health and medicine can help people better achieve their own health and eating goals, what would that be like? That’s a thought….

13 thoughts on “What do weight-loss studies REALLY show? Looking behind the scenes

  1. This is a really terrific critical piece. I don’t see your answer to the title question, though: given all of these methodological weaknesses, what conclusions should we take away from this body of research?

    1. Hi Dan– thanks for the kind words. And Oops! Yes, you’re right. Of course the answer is complicated (as you know). We can conclude: 1) we don’t actually know which (if any) and to what extend any weight-loss plans are effective, because we don’t do the work to track people over the long-term (five years is standard tracking for most medical treatments). 2) The medical establishment’s views about the efficacy of dieting are so entrenched, that even faced with such dismal results, they STILL say that they would recommend some of these diets, as they think 3–5% weight loss (provided that’s even a reliable estimate, which there’s reason to think there’s not) is “a good start”. Which is not the start, but instead the best clinical endpoint the studies purport to show. Okay, enough for now– we should talk more at the conference in Dallas!

  2. That’s the first time I’ve seen a meta-analysis assess the potential for bias (beyond pointing at research from the 60’s and shouting andro/ethnocentrism) in a “scientific” study, but perhaps it’s just not popular in my fields. If more studies did this, we could have more meaningful conclusions from all areas of science.

  3. We may not like the answer, but it seems pretty clear that it’s just what you say: diets don’t work. The only thing that will change our weight (or, for that matter, anything else we want to change for the long run) is a permanent change in our lives. I am speaking as one of the lucky few who have managed to maintain a significant weight loss for many years (35 pounds, about 25% of my starting body weight, for 12+ years) so I know where it came from : 1) I was a skinny kid, teen and young adult, so I know my genes help; 2) I stopped eating when I was not hungry, or eating more than I needed to feel comfortably full – that’s the only form of “dieting” I ever did; 3) – this is the big one – I turned from a modestly active person who walked a fair amount, to a fairly hard-core athlete because I fell in love with cycling. I was 49 when I started the process. I want to add that I do NOT think this is something everyone should feel obligated to do. Yes, I feel great and energetic, but only because this phase of my life is lots of fun for me. I could also have been perfectly happy as I was before. So anyway, when these systems stop making unsustainable promises and start helping us all just plain feeling happy and well, whatever that means to us, I’ll start respecting them. Until then…they’re just commercial endeavors with a strong interest in keeping customers for as long as they possibly can.

    1. Great comments, Winnie! And yes, many of us agree that our health systems should help us to feel happy and well, not guilty and inferior. Congrats on the life changes and happiness; in general it’s hard to sustain big changes, and one should do a dance when it sticks… 🙂

  4. My former boss plans to become a Weight Loss leader in her early retirement. She herself lost over 100lbs. and so far has managed to keep it off for last 2 yrs. or so. She also adds exercise..

    From little that I hear about it, it would drive me insane to consider points value on each type of food. Not sure how that helps to intuitively eat balanced meals daily.

    May I suggest that in terms of just walking building it into your daily lifestyle: in long term care facilities, there’s some signage all over the place to walk everyday..most health care professionals recommend to those who are frail. I saw it in hospitals in Toronto when my father was dying. Even if it’s very slow movement.

    You know what super frustrated him: he could no longer walk at all. (He had cancer…but no other problems except for some spinal stenosis.)

    Don’t wait until then….rejig your lifestyle..now. Don’t take your mobility for granted and take advantage of it now.

    1. HI Jean– yes, your advice for all to keep walking is good; for those with disabilities, life is more complicated, but we deserve a health care system that helps everyone find sustainable ways of living in happy and mostly pain-free fashion. Here’s hoping…

    1. Hi– I’m not a medical professional, but as an obesity researcher I can say that the notion of “gaining control of eating” is really complicated, and we don’t have complete control; we are subject to genetic, deep psychological, cultural, and advertising influences, and we have a lot of constraints on us: economic, family and culture, our own biology, our own psychology, etc. What a lot of folks (especially on this blog) would like to see is more help for us to eat in less burdened ways that make us feel good and good about ourselves. And people can do that at all different weights. Finally, there’s lots of evidence that diets undermine rather than support our efforts to eat in those more beneficial ways. Keep reading this blog, and keep posting– the discussion is important!

  5. You wrote:
    “Attrition: Attrition is easier to measure– you just count up who is still in the study at the end. Right. But again, like in the adherence case, we want to know WHY people drop out. The same intriguing questions remain, but these studies aren’t designed to ask them. Why not? Beats me.”

    Speaking as a researcher who conducts randomized controlled trials (though in a different field), I can tell you that we often can’t ask people why they drop out of studies/don’t show up for final measurements because we don’t know where they are. They move, for example. Some populations are harder to follow up with than others. However, you would routinely expect that people who agreed to be in a weight-loss study would also have provided multiple ways to contact them, so high levels of attrition are a problem. Just because someone quits Weight Watchers or Jennie Craig doesn’t mean they should disappear from the study, and a well-designed and well-implemented study should absolutely include those people in the final outcomes.

    I am very happy to see someone who is not a researchers grappling with the meaning of meta-analyses. I think we all need to be better informed by scientific studies and driven less by sensationalist reports of the rare person who lost 100 pounds on xyz diet. Thanks for your blog.

  6. Great, great, great post! I cringed when the Today show reported about the “great results” for Jenny Craig and Weight Watchers. We not only need to understand the data better, but prevent the media from reporting it incorrectly. A member of my family is a cyclical dieter and this type of media promotion of a “research study” just encourages her to not make the real life changes that are holding her back from living a healthy lifestyle.

Comments are closed.