John P.A. Ioannidis Hits Another Home Run on Epidemiological Research


(Central Florida Bob ) #1

John P.A. Ioannidis is the author of what’s widely quoted as one of the most downloaded papers in history, “Why Most Published Research Findings are False”, in which he presents data that as much as 70% of published science is wrong.

In an article in the Journal of the American Medical Association (JAMA) last year, he took on nutritional epidemiology. Bottom line, this field has got to be fixed because it is just so far from what’s considered science that it’s dangerous. Not only is it endangering peoples’ health, it’s ruining confidence in science as a way of finding out how the world works. You can read or save the paper (2 page pdf) here.

Some quotes to get you to read it.

In recent updated meta-analyses of prospective cohort studies, almost all foods revealed statistically significant associations with mortality risk.1 Substantial deficiencies of key nutrients (e.g., vitamins), extreme over consumption of food, and obesity from excessive calories may indeed increase mortality risk. However, can small intake differences of specific nutrients, foods, or diet patterns with similar calories causally, markedly, and almost ubiquitously affect survival?

One of my favorite paragraphs, with some emphasis added.

Assuming the meta-analyzed evidence from cohort studies represents life span–long causal associations, for a baseline life expectancy of 80 years, eating 12 hazelnuts daily (1 oz) would prolong life by 12 years (ie, 1 year per hazelnut),1 drinking 3 cups of coffee daily would achieve a similar gain of 12 extra years,2 and eating a single mandarin orange daily (80 g) would add 5 years of life.1 Conversely, consuming 1 egg daily would reduce life expectancy by 6 years, and eating 2 slices of bacon (30 g) daily would shorten life by a decade, an effect worse than smoking.1 Could these results possibly be true?

No they can’t be true. They’re a result of the way these meta-analyses work; they find spurious correlations.

Individuals consume thousands of chemicals in millions of possible daily combinations. For instance, there are more than 250 000 different foods and even more potentially edible items, with 300 000 edible plants alone. Seemingly similar foods vary in exact chemical signatures (eg,more than 500 different polyphenols). Much of the literature silently assumes disease risk is modulated by the most abundant substances; for example, carbohydrates or fats. However, relatively uncommon chemicals within food, circumstantial contaminants, serendipitous toxicants, or components that appear only under specific conditions or food preparation methods (e.g., red meat cooking)may be influential. Risk-conferring nutritional combinations may vary by an individual’s genetic background, metabolic profile, age, or environmental exposures. Disentangling the potential influence on health outcomes of a single dietary component from these other variables is challenging, if not impossible.

The JAMA article by John Ioannidis is a good read, but it’s a scientific/medical paper and a bit dense for people who aren’t roughly familiar with the statistical methods. I found the link to it on Watts Up With That, a weather/climate science website and the author there used the paper to draw links between the problems in nutritional epidemiology and the problems of climate modeling.

His article may be more readable.


#2

@CFLBob, I also find this sort of thing interesting to read, although admittedly challenging sometimes. Thus far it’s been Peter Attia’s “Studying Studies” which has influenced my general skepticism. I’d think any of these, however, would be time well spent for most keto-ers, just to help keep the scare stories in perspective!
Thanks for calling this out!


(Central Florida Bob ) #3

Thank you, @Annoula, for pointing out “Studying Studies”! I don’t go to Peter Attia’s pages all that often and see he’s been up to a lot since the last time I visited.


#4

@CFLBob, my pleasure of course. And here’s another article far easier to read/absorb – https://www.theguardian.com/society/2016/apr/07/the-sugar-conspiracy-robert-lustig-john-yudkin. It will be old news to any number of keto-ers I’d guess, and it doesn’t go deep into the science, but I re-read it recently and was impressed again how well it does cover in general terms why one should be very skeptical of mainstream “science.”
(It was this very article, BTW, that started me on my keto ‘journey’.)


(Bunny) #5

Here is a recent e-mail by Dr. Peter Attia I received yesterday that kind of illustrates those points:

Greetings -

Want a real-life example of how difficult it is to make inference and why science and the scientific method are essential if we value the advancement of knowledge? (Borrowed from an Instagram post last week.)

For the sake of illustration, I’ll use an example from my recent archery practice, though this point is not about archery, but about inference and the process of hypothesis generation, and testing.

I closed my archery practice with the following set:

  • 10 rounds of 6 shots at 50 yards onto a 65 cm target.
  • Scoring is as follows: 5 points in the white, 4 in the inner black ring, and 3 in the outer black ring. If you hit the “x” (i.e., the inner white ring) you get the 5 points, but also an “x” to break ties.

Take a look at the result of my first round. My green scored 15, 3x and my pink scored 13, 0x:

I thought, “Man, there must be something wrong with the pink fletched arrows!”

Point #1 —we always look for patterns

Pattern-seeking behavior help animals, and us humans, survive. (It also helps us go through the day without having to make predictions about every moment of every day.) If it seems like a subtle movement detected in the bushes sometimes translates to a predator behind those bushes, our first thought is not to run an experiment and see if we get eaten.

Then I said, “Wait, that’s too small a sample, let me do an ‘experiment’ and see how it shakes out after all 10 rounds.” (A fair effort, but really the only thing I could do that was experimental was randomly select the arrows from my quiver, which I did, to be sure it wasn’t an artifact of something like fatigue or shot sequence.)

After a few rounds, it really seemed like the green fletched arrows were flying better, but in my mind, I said, “No way…I know they are the same!”

Point #2 —still biased

Here, I’m having an internal argument, and it actually may be somewhat competitive. That is, my bias is that there’s no difference between the arrows. There may be a Rosenthal effect at play here: I may subconsciously behave in a way that somehow tips the scales in favor of my preconceptions.

“I fletched them myself (i.e., I was the one who attached the green and pink fletchings to the shafts of the arrows) and there is no way color alone could account for this difference.”

Point #3—this is a generally accepted and good idea in science, to maintain a strong null hypothesis, but if it impacts your ability to be objective (see below) it’s a problem?

Then I found myself ‘trying harder’ with the pink arrows to narrow the gap between them!

Point #4 —without blinding, it is damn near impossible to eliminate experimenter bias

This is a variant of the Hawthorne effect. It’s also sometimes referred to as the Avis effect, after the rental car company’s slogan, “We try harder.” The aforementioned Rosenthal effect is named after a researcher’s study that supported the hypothesis that reality can be positively or negatively influenced by the expectations of others. Rosenthal argued that biased expectancies could affect reality and create self-fulfilling prophecies.

In this case, I’m the observer and the subject in the experiment. That is a dangerous combination. I’m trying harder in the direction of confirming my preconceptions. I should be asking: does my narrative fit the evidence? However, I’m the one producing the evidence, and I may consciously or subconsciously manipulate the evidence. I may be creating a self-fulfilling prophecy.

In the end, the results were (far from stellar, by the way, even at this distance):

  • Green shot 138, 8x
  • Pink shot 133, 6x

So, all things considered, I have no idea if there is a difference between the green and pink fletching. But that’s not the point of the story.

We’re emotional creatures that look for signals in a sea of mostly noise. We like to see things as we wish them to be and we sometimes consciously or unconsciously act in a manner to coax those things to our wishes. Without a Ulysses pact, fooling ourselves appears to be the default state of the human condition. Without a framework, in this case, the scientific method, we’re far too likely to see what we want to see rather than the alternative.

This story serves to remind me that we are not wired to think scientifically. Full stop. It is the quintessential human flaw. But scientific thinking is a skill to be practiced and improved upon. I’m better at it today than I was 5 years ago, and I’m confident 5 years from now I’ll be saying the same things. So can you.

  • Peter

(Bunny) #6

I think is safe to say (being objective?) we have three distinct camps?

• Vegetarian and various offshoots that focus more on nutrition from plants? (the majority are not really observing ketone body measurements maybe out of fear of promoting the Ketogenic diet maybe that’s where the self-fulfilling prophecy takes root?)

• Ketogenic Dieters and various offshoots that focus on dietary proteins and fats but more so from animal sources?

• …and those that eat unlimited whatever?

Within the above two distinct camps they both reverse diabetes but are they really different? NO

They both do the same thing, they both produce ketone bodies when the body quits doing it, as a result of being in the camp of the unlimited eat whatever?

Being omnivores we can do the same thing the gut of an herbivore animal can do, not entirely; but we can also produce butyrate?

Or we can eat the fat and protein of the herbivore animal (grass fed; raw milk & butter etc.) to get the butyrate and get the same result?

We can also eat more fat and protein and keep the carbs under 50 grams a day to make the mitochondria favor ketone bodies rather than glucose?

This is a good example of bias; are we creating self-fulfilling prophecies by pointing fingers to condemn the other camp? (even at the eat whatever camp?)

A person like myself could easily eat anything but not as much and place myself in ketosis and be in the eat whatever camp, but who would believe that?


(Mame) #7

As an epi I would argue that it is impossible to ‘fix’ the field of nutritional epidemiology in any ethical way. it’s using the wrong tool entirely. but that’s just my opinion. :joy:


(Central Florida Bob ) #8

Wonderful posts, @atomicspacebunny! Have you read Denise Minger’s Death by Food Pyramid? It’s an attempt to reconcile the two camps. That’s hard to do, and I’m not sure I’m doing her sufficient justice, but she concludes the essential commonality between healthy vegetarians and healthy omnivores is avoiding the junk, processed foods. Things like seed oils, excessive sugar in everything and so on.

In my “checkered past” I studied biochemistry through the beginning of my senior year of college and then (due to a combination of bad luck and bad decisions) had to quit and start working for a living. I turned my childhood hobby of playing with radios into a career as a technician and then a design engineer - including starting college over and graduating. The advantage of this is that I had tons of training in doing controlled experiments and what I’ll call the real scientific method. Along the way, I took two statistics classes, one undergrad and one grad level. Those helped with understanding the way correlation studies work. When I started really paying attention to nutritional epidemiology, I mean besides just hearing the endless “eat this, don’t eat that” stuff, I was appalled.

The perfect example is Brian Wansink, the king of junk food science, where the adjective “junk” modifies science, not food. If you want an example of how science should not be done, it’s hard to do better than the pizza restaurant example in this article from February of '18.

By September of '18, Wansink was out at Cornell. I don’t know if he’s working in the field now, but he was so fast and loose with his statistics that I don’t see how he could be.

The political site FiveThirtyEight did a good summary of how he searched for spurious correlations, and included some correlations they found themselves. It’s good for a laugh.

What I think I see emerging is something people don’t like thinking about and the “food establishment” certainly wouldn’t like: we really don’t know anything. Virtually ALL of those “eat this not that” studies we’ve heard all of our lives, and that people joke about now, are useless. Throw them all out: high carb/ low fat (Food Pyramid), Mediterranean Diet, you name it. All of it is indistinguishable from junk science. Right now, I think the only real science is pretty small, like the Virta Health studies.

Dr. Ioannidis points out that with “more than 250 000 different foods and even more potentially edible items, with 300 000 edible plants alone. Seemingly similar foods vary in exact chemical signatures (e.g., more than 500 different polyphenols)” the situation is so complex that controlled experimentation is pretty much impossible. Statistical analysis of variation (ANOVA) provides some hope, but if there’s more than a handful of things to track, forget it as well. The final nail in the coffin of randomized controlled trials for foods is that to be meaningful, they’re so expensive that they simply can’t be done.

This is an old joke, but I see it’s relevant again today.

But the joke tells an important truth, too. If one study says a food is bad while another says it’s not, and that pattern continues for years, it might be that the effect is so small it’s not worth thinking about.


(Bunny) #9

Yeah I think the egg argument is a nice gauge of an example by creating public distrust almost like they are hiring people to manufacture this type of falsified (only tell part of the truth?) media propaganda as the sugar producers fear people will quit buying their stuff?

Will it be bad if you throw a whole bunch of sugar on top of those eggs?

So it is a nice argument to make just don’t mention sugar part of it so they will believe you?


(Central Florida Bob ) #10

Things that they never talk about.

I thought Stanford did the ultimate randomized controlled trial on eggs something like 20 years ago. Took two groups, kept them in a diet ward (so it was a short duration - month long, IIRC), fed one group eggs and the other group something else. Halfway through, they measured a bunch of blood markers (probably meaningless stuff like cholesterol profiles) found no significant difference and then crossed the groups so that the previous egg eaters ate no eggs and the second group now got the eggs. No difference between the groups again.

Simple, elegant, and the only thing you can argue against it is that it wasn’t hundreds of thousands of subjects monitored for decades.


(Bunny) #11

In that case I would say a ten year sample would be sufficient? (It will not change even if you took a 20 year sample?)


(Bob M) #12

You know, it’s too bad Peter Attia doesn’t follow his own advice. He simply refuses to believe that LDL (or LDL-p, or whatever the darling marker is now) does not CAUSE heart disease. Even though all the evidence is epidemiological for humans. He’s one of the people I ceased following for this reason, among others.

Anyway, Ioannidis is a rock star. They tried to get him onto the committee for setting nutritional guidelines in the US, but no one wanted to rock the boat that much.