I just found out this week that US News and World Report, the now-online-only news magazine that ranks everything from universities to hospitals to cruises (I’m not kidding–check out your cruise rankings here), has also been in the diet ranking business for the past 8 years. Where have I been all this time?
Well, even though I’m late to this party, I decided to dive in and see what was up (I’m leaving the very mixed metaphors in for flavor). Warning: their diet rankings page, found here, can send you down the internet rabbit hole. I started nosing around, and before I knew it, two hours had passed. Yes, it was like this:
But I did find out some interesting things, which I will now share with you. I’ll put them in the form of an FAQ:
- How did US News and World Report go about ranking diets?
Here’s what they said:
To create the eighth annual rankings, U.S. News editors and reporters spent months winnowing potential additions to our diet roster and then mining medical journals, government reports and other resources to create in-depth profiles for those that made the cut.
Yeah. I’m sure they spent a bunch of time on this. But I don’t know what an “in-depth profile” is. It is almost certainly not something that meets scientific standards, which is a meta-analysis.
What’s that? Here’s Wikipedia:
Conceptually, a meta-analysis uses a statistical approach to combine the results from multiple studies in an effort to increase power (over individual studies), improve estimates of the size of the effect and/or to resolve uncertainty when reports disagree.
Want more detail? Here’s more precise info from a medical journal discussion on meta-analysis:
Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess previous research studies to derive conclusions about that body of research.
Outcomes from a meta-analysis may include a more precise estimate of the effect of treatment or risk factor for disease, or other outcomes, than any individual study contributing to the pooled analysis.
The examination of variability or heterogeneity in study results is also a critical outcome.
The benefits of meta-analysis include a consolidated and quantitative review of a large, and often complex, sometimes apparently conflicting, body of literature.
The upshot of the above is this: if we really want to know the answer to a scientifically complicated question like “which diet is the best for X”, and we know that many scientists have explored this question from many angles, getting many different answers to parts of this question, then we need a systematic way to analyze all those studies and combine them to try to get a coherent and reliable answer to this complicated question. It is not a foregone conclusion that doing a meta-analysis will end up yielding either a coherent or reliable answer. The data may be too conflicting, or there may be gaps where we don’t know enough, so we can’t draw conclusions. This is all useful for researchers, as it points to areas for further study. However, it may or may not help clinicians who want answers for their patients, or the rest of us, who just want answers.
2. Wait a minute– they didn’t consult actual experts in medicine, nutrition, etc.? How can they assess loads of technical scientific information without help?
Here’s what they said:
A panel of nationally recognized experts in diet, nutrition, obesity, food psychology, diabetes and heart disease reviewed our profiles, added their own fact-finding and rated each diet in seven categories: how easy it is to follow, its ability to produce short-term and long-term weight loss, its nutritional completeness, its safety and its potential for preventing and managing diabetes and heart disease. We also asked the panelists to let us know about aspects of each diet they particularly liked or disliked and to weigh in with tidbits of advice that someone considering a particular diet should know.
Yeah, let me unpack this for you. US News and World Report gathered together a panel, which includes clinical practitioners in medicine and nutrition, some diet book writers, directors of clinical programs for diabetes, heart disease, etc., and researchers and clinicians in academic medicine. Some of these folks have a particular diet they’re promoting, and others are very busy either running a medical center or doing research and running a lab.
At best, some of these panelists read the pre-prepared packets of material on various popular diet plans (like the Mediterranean Diet or the DASH Diet) and offered their best guesses, based on their areas of expertise. More likely, the busiest experts farmed this task out to grad students or research assistants, which is legitimate and common practice.
3. So far, all this sounds okay. What’s the problem?
Remember what I wrote (which you may have skimmed over, which is perfectly fine) about meta-analysis? I said that to assess a program like a diet, you have to do very fancy statistical work on dozens–hundreds of studies on that program or diet. It takes a lot of time and brain power and qualified people, and results in a journal article. US News and World Report couldn’t do this, and the expert panels didn’t do this. So they’re just doing the equivalent of throwing darts at a dart board, in my opinion.
This procedure is not the way I would choose to make decisions about my health.
There’s more to say (as always), but I’ll leave it at this: don’t let the people who advise you about what cruise to take also tell you how to eat in order to deal with your diabetes. Just sayin’.
