“Butter is back,” declared an op-ed by food columnist Mark Bittman in The New York Times in March 2014.
“That the worm is turning became increasingly evident a couple of weeks ago, when a meta-analysis published in the journal Annals of Internal Medicine found that there’s just no evidence to support the notion that saturated fat increases the risk of heart disease.”
Perhaps that was a reasonable conclusion for a food writer to reach. After all, a meta-analysis that combines the results of many (in this case 72) studies sounds like a sweeping review of the evidence. And a meta-analysis can be valuable…if the authors know what they’re doing.
“Almost anyone with an Internet connection can do a meta-analysis,” says Walter Willett, who chairs the nutrition department at the Harvard School of Public Health.
“It’s just pulling data out of the literature. It has the impression of being comprehensive and complete, but it can cause a huge amount of damage.”
And the problem is getting worse.
“There are groups of professional meta- analysis people who believe that you shouldn’t know anything about the topic you’re investigating because that will cause bias,” Willett explains.
“That may work for drug trials where it’s just pill versus placebo, but for something complicated like human disease and nutrition, it’s important that you know what you’re doing or else you’ll do some very silly things.”
The March meta-analysis on fats and heart disease was a perfect example.
“The lead authors have done some good work in the past, but they really didn’t know the topic,” says Willett. “They corrected some of the gross errors after the paper was published, but it still had major omissions and layers of problems.”
One example: the authors concluded that people who were randomly assigned to replace saturated fats with polyunsaturated fats had no lower risk of heart disease. That contradicted an earlier meta-analysis. Why? Because the new meta- analysis added a trial in which people in the polyunsaturated group were given a margarine loaded with trans fat, which raised their risk of heart disease.
“If you look at the trials that replace saturated fat with fats like soybean oil, which have both omega-3 and omega-6 fats, those actually show a lower risk of heart disease,” says Willett. “But that was buried in the supplementary tables online and ignored in the paper itself.”
What’s more, the new meta-analysis said not a word about evidence showing that saturated fat raises LDL (“bad”) cholesterol and that polyunsaturated fats lower it.
“There’s 50 years of work on lipids that the authors didn’t even bother mentioning,” notes Willett. “Those data make it very clear that the type of fat makes an important difference.”
Instead of a meta-analysis, which combines each study’s final results, it’s far better to pool the original data from each study. That’s almost as good as having data from one large study, but it takes far more time and expertise than a meta-analysis.
“In 2009, we pooled the original data from 11 studies—three of which were omitted from the recent meta-analysis,” says Willett. “We showed that replacing saturated fat with polyunsaturated fat is clearly related to a lower risk of coronary heart disease, yet the paper got almost no press.”
In contrast, the recent man-bites-dog, comprehensive-sounding analysis made headlines around the world.
And it’s not just fats and heart disease. Several meta-analyses have reported that eating less salt doesn’t prevent—and may even cause—deaths from heart attacks and strokes.
“The salt industry has been paying people to do meta- analyses on salt for years,” says Willett. Among their flaws: including people who eat less salt—less food, really—because they are already ill.
The scientists who review the evidence should know what they’re talking about.
“It’s an abuse of a meta-analysis, because you check your mind at the door and just mechanically throw in numbers,” says Willett. “A meta-analysis can provide a valuable summary when it’s carefully conducted and the number of studies on a topic becomes large, but at present we have a meta-analysis plague, and it’s dangerous.”
Sources: Ann. Intern. Med. 160: 398, 2014; PLoS Med. 2010. doi:10.1371/journal.pmed.1000252; Am. J. Clin. Nutr. 89: 1425, 2009.