Beware the Man of One Study, or be cautious with the evidence

#1

This is a bit Meta for the Show me the Science category, as it’s more about both why we should look at the science and why we should still be cautious with the science, whether it favors our ideas or not (which I think we’re good about around here to some extent at least).

I’ve frequently thought about this blog and the basic thesis of it often when considering nutrition debates and even my own attitudes and research on nutrition and the ketogenic diet in particular (a bit of a caution as this particular blog is not about nutrition though it does deal a bit with medical studies, but also goes a bit into particular political topics. It isn’t trying to press any particular side in this to a large extent and they are just examples for the point):
Slate Star Codex: Beware the Man of One Study

But I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.
Which is too bad, because that’s exactly what people who want to bamboozle an educated audience are going to use.

Now, I can say I’ve done a decent bit of research on various dietary topics, and even early on in looking into Keto I decided I didn’t trust an echo chamber and so looked around and listened to what other routes had to say to support themselves and against a ketogenic diet, etc. Still, I’m no expert in the field by a long shot, and I can’t really say I have anywhere near a complete survey of the field. Could I be taken for a loop here? Are other sides being taken for a loop? Is the matter simply not settled? These are certainly questions to consider and reasons to be cautious with any individual study we see, as one study, one experiment, isn’t super meaningful. It needs to be confirmed over and over, and even then one should expect some instances where things came up in a way that is contradictory (flukes, badly designed experiments, set of mutants involved, was looking at something slightly different, etc).

#2

Well, I really like getting meta-cognitive…

Indeed, critical thinking is essential, and not just that - critical thinking that is emotionally intelligent, integrative in its analysis, and cross-culturally informed. I would personally add humble as well - because hubris (and the doctrines or belief that ethics are passé, that there is little social and ecological responsibility) is very destructive, as we can see by looking at what industrial agriculture and the cultural genocide of indigenous peoples has done to the living earth and human culture.

I think conventional curricula and mainstream experts often function to reproduce the culture of cartesian dualism that separates everything and everyone into parts, and denies the great mystery that the sum of the whole is greater than the parts, and that we are in relationship with non-linear equations, nuances, spiral dynamics (systems theory) and culture. It is on a quest to conquer all objective evidence and minimize or displace empirical evidence. So, what may be evidence may not be true intelligence, if the values that inform the evidence are “profits over people”, or “using rather than stewarding” or “profitably & temporarily curing rather than durably healing through lifestyle & food-as-medicine” etc. And if what is evidence is not an industrially profitable evidence (such as the reality of cellular healing and cancer remission rates) it’s frequently derailed or just ignored, rather than incorporated and expanded upon for the greater good.

Am a perpetual student of both my own experience and valuing both the empirical and objective - as well as those from well-vetted mentors and independent researchers who I respect. An important piece here is that whether the man is The Man of One Study or Many Studies, The Man is also the euphemism for the hierarchically exploitative male-bias of patriarchal civilization (as opposed to 90%+ of human history, which was egalitarian). :muscle:t3: It’s a grave matter when it comes to women’s health, human birth & neonatal practices, and, increasingly, biology in general - as many universities take a biotech perspective on biology now.

I think the geopolitical changes that happened in the 1990s with world trade has seen scientific study become a marketing battlefield, with biases of industrial culture itself accepted as objective reality, its own doctrine. Context maters. What is “evidence-based” is not necessarily what is true, or whole, in such a context.

So, being that western industrialism and its scientific method in medicine is very young (geez, doctors didn’t take germ theory seriously for years, and didn’t even start washing their hands until the late 1850s, after tens of thousands died due to this fact) and that it is founded in colonialism and cultural genocide - I think it has an inherent disregard for even its own early european roots, such as hygiene practices of ancient traditions, as well as the Hippocratic method: first, do no harm.

And, in capitalist countries without single payer healthcare (as opposed to a number of socialist-leaning countries such as the Nordic nations, and France etc) profits are being made hand over fist on prescriptions and surgery - along with the unscrupulous funding of studies by large corporate interests (much of the time, pharma companies or industrial agriculture). This, then, certainly influences what is asked by those paid to study - and what “the evidence” is, as well as how findings are reported, and what money can be made off of such reports.

What’s true for me - and true from the sub-atomic level to the cosmos - is that absolute and the relative are always in relationship and at great rapidity :smile: Truth is a rare commodity in post-postmodern industrial culture, and I look for confirmation that contains moral & ethical integrity and cross-cultural respect in its confirmations. I especially like the bioregional food/herbs-as-medicine realm of health & healing (paleoanthropology, cultural anthropology - and the work of Nora Gedgaudas, Ron Schmid, and Sally Fallon Morell in the book Primal Nutrition: Paleolithic and Ancestral Diets for Optimal Health. Gedgaudas, along with many other astute and wise female voices, was featured in the recent FAB Australian ketogenic documentary called The Magic Pill, slammed by the Australian Medical Association - which means probably the American and British medical associations too - because it’s THAT good.

(Rob) #3

I find the science behind keto to be unambiguously compelling and much of the counter points to be biased and poorly constructed - but then I would say that - I’m a believer :grin:

Being skeptical is vital - it is the pathway to believing in keto. You have to be skeptical of the status quo but it is not hard to see the overarching logic. We have never eaten less saturated fat as a population nor more carbs (of all kinds), yet have never been sicker. That says something is very very wrong with the status quo. Then you have to try to pick an alternative solution. Vegan/plant-based isn one way to go. Any number of fads in between all the way to keto to zero-carb/carnivore are other alternatives.
You then need to investigate what you want to. You can look at the China Study to support veganism but then you HAVE TO push through the rebuttals, the rebuttals of the rebuttals and the rebuttals of the … you get the idea. Then you have to decide what you believe. If you are being open, you probably have to do that for each major dietary approach and each fad in between. Otherwise you need to try what you believe and see if it works.

The strength of keto is not in the science… IT IS IN THE PRACTICE. Look around these forums for all the evidence I will ever need… at least to try it properly. I have tried it and have been amazed by my own results. Weight loss, lower blood pressure, mental clarity etc.

All this said, it is ALSO vital that you understand context. For instance, veganism (sensibly executed and properly supplemented) is healthy and sustainable for SOME people. Knowing someone’s starting point is critical to knowing what might work. If you are severely insulin resistant, have metabolic syndrome, etc. then any diet is better than SAD but I’m convinced that Keto is the best way to go. Once you have critically assessed the science around saturated fat and cholesterol (via Taubes, Teicholz, Feldman etc.) then Keto looks safe and the overwhelming numbers of real success stories every day here and all over the LC community suggests the same. If you are metabolically healthy, go vegan. Good luck to you. I would find it impossible but everyone is different.

So be skeptical but then choose what YOU believe in. Then continue to be skeptical but based on an increasing solid knowledge base it will get easier and more rewarding rather than be paralyzed by analysis and doubt. Obviously, be prepared to evolve your knowledge base when you see something convincing you to change a view point but confidence is empowering.

#4

I find n=1 experiments extremely useful, because what I’m most interested in is what works for my body specifically. And for many things, that can only be determined by personal trial, tracking and adjustment. Reading about how other people approached an issue, designed a personal experiment and their results helps me formulate mine and provides context for interpreting my results.

Observational studies are important because they identify correlations. These can then be tested to determine if the relationships are causal or merely associative. This is especially important in instances where a detrimental impact may take years to manifest.

When it comes to nutrition, I don’t think studies will lead to a single path that is optimal for all (or even most). I think our genetic diversity, environmental differences, and lifestyle inevitably lead to a mutifactorial solution.

(Michele) #5

In addition n=1 may give us a certain result today and in 6 months time that n=1 may be different due to the earlier outcomes of n=1. This is why n=1 is so important. What works for me in the nitty gritty finer detail may not work for another today, but might do in the future or not at all. n=1 is about taking personal reaponsibility particularly when we are taking close note of the outcomes and what we are doing to contribute to outcomes (and also being aware that the relationship may not be causal in some instances).

#6

I feel like there may be some confusion about what the point being made with the linked article is. It’s actually not considering n=1 experiments at all (those do not make it into “studies” or into peer reviewed journals and meta-analysis). The ideas may be somewhat applied there, but it’s really talking about how you consider these larger studies, being aware that even larger studies will show different results between them and cherry picking studies can be used to deceive or skew towards a view the presenter wants regardless of merits. Thus, caution should be used when listening to claims of “study such and such finds” or “we know this because this analysis showed that”, etc. (even if such things are better than not having anything).