Categories
Clinical trials Complementary and alternative medicine Medicine Paranormal Pseudoscience Quackery Skepticism/critical thinking

Teaching critical thinking to combat fake news and bullshit: You have to start young

Thanks to social media, fake news, conspiracy theories, and health scams spread faster and farther than ever. The world is in need of critical thinking skills now more than ever. Fortunately, there is hope. Critical thinking can be taught, but teaching these skills works best if you start young.

As much as I like to deconstruct pseudoscientific claims, particularly about health, medicine, and health care, Sometimes it gets a bit draining. There’s just so much pseudoscience, so much credulity, so much sheer idiocy out there that trying to refute them and encourage a more skeptical mindset often feels like pissing into the ocean, for all the effect it has. In the age of fake news and Donald Trump, it even feels as though we’re going backward—and not slowly, either. That’s why I felt it was time for a bit of a break, a bit more optimism than I’ve been able to muster before. So it was a good thing that I happened across an article by Julia Belluz and Alvin Chang entitled This researcher may have discovered the antidote to health bullshit.

I’ve always suspected that the key to combatting bullshit of all stripes, be it related to health (one of the more dangerous forms), conspiracy theories, or one of the many other varieties of nonsense, pseudoscience, conspiracy theories, and quackery, will be starting young. Let’s face it. By the time we’re adults, changing the way we think is very difficult. That’s not to say that it can’t be done, but in general you have to want to change. Again, let’s face it. Most people don’t want to change. They resist it. That’s part of the reason why adults are so good at motivated reasoning, in which they become very good at finding observations and evidence that support their preexisting point of view while downplaying or discounting evidence that does not. In essence, they develop only part of the skills needed to be a skeptic in that they are very good at deconstructing ideas they disagree with but remain not so good at critically examining ideas that they do agree with that might lack good evidence to support them.

Yes, as Andy Oxman shows, you have to get ’em young. He relates a story about his visit to his then 10 year old son’s class in 2000.that shows how children actually have a proclivity for becoming skeptical:

“I told them that some teenagers had discovered that red M&Ms gave them a good feeling in their body and helped them write and draw more quickly,” Oxman said. “But there also were some bad effects: a little pain in their stomach, and they got dizzy if they stood up quickly.”

He challenged the kids to try to find out if the teens were right. He split the class into small groups and gave each group a bag of M&Ms.

The children quickly figured out they had to try eating M&Ms of different colors to find out what happens, but that it wouldn’t be a fair test if they could see the color of the M&Ms. In other words, they intuitively understood the concept of “blinding” in a clinical trial. (This is when researchers prevent study participants and doctors from knowing who got what treatment so they’re less likely to be biased about the outcome.)

In a short time, they were running their own blinded, randomized trials — the gold standard for testing medical claims — in the classroom. By the end of their experiment, Oxman said, “They figured out that there was little if any difference in the effects of the different colors and they asked me if the teenagers who made the claim really believed that.”

As a result of this experience, Oxman began working with other academics to develop curricula designed to teach critical thinking skills in children. The idea is to arm them with the skills needed to spot “alternative facts” more readily and, as importantly, how to keep fake news from spreading in the first place. He’s not the only one who asked this question. So did Imogen Evans, Hazel Thornton, Iain Chalmers and Paul Glasziou, who wrote a book called Testing Treatments that’s available for free as a downloadable PDF. In 2012, Oxman teamed up with one of the authors of the book, Sir Iain Chalmers, asking him to help to develop curricula based on the concepts in the book appropriate for primary school children

Because Oxman had ties with researchers in Uganda, he tested the new materials there on children between 10-12 years of age. Personally, when I first saw that, I thought that ten years old is a bit old to start the process. I would have thought that age eight or even six would be the time to start. Be that as it may, Oxman ran a randomized trial in which a representative sample of eligible schools were randomized either to control (no change in curriculum) or including the Informed Health Choices (as the program came to be known) primary school resources (textbooks, exercise books, and a teachers’ guide) in the lesson plan. Teachers teaching the Informed Health Choices curriculum attended a 2 day introductory workshop and gave nine 80 min lessons during one school term. The lessons addressed 12 concepts essential to assessing claims about treatment effects and making informed health choices.

Twelve key concepts were emphasized:

Claims

  • Treatments might be harmful
  • Personal experiences or anecdotes (stories) are an unreliable basis for assessing the effects of most treatments
  • Widely used treatments or treatments that have been used for a long time are not necessarily beneficial or safe
  • New, brand-named, or more expensive treatments may not be better than available alternatives
  • Opinions of experts or authorities do not alone provide a reliable basis for deciding on the benefits and harms of treatments
  • Conflicting interests may result in misleading claims about the effects of treatments

Comparisons

  • Evaluating the effects of treatments requires appropriate comparisons
  • Apart from the treatments being compared, the comparison groups need to be similar (ie, “like needs to be compared with like”)
  • If possible, people should not know which of the treatments being compared they are receiving
  • Small studies in which few outcome events occur are usually not informative and the results may be misleading
  • The results of single comparisons of treatments can be misleading

Choices

  • Treatments usually have beneficial and harmful effects

These are indeed key concepts that any medical skeptic needs to know and understand. I’ve empnasized pretty much all of them at one time or another over the last 12 years. Heck, we even have names for some of them, such as the appeal to antiquity to describe the concept that just because a treatment is old does not make it better. After all, many of these “ancient” remedies date back to times when medicine was anything but scientific and the very basics of what causes disease were not understood and instead diseases were attributed to “imbalances” in humors, “bad air,” or even the intervention of malign spirits. Any of them that might work were basically discovered by sheer accident, and ancient herbal remedies that work have mostly already been picked over and turned into purified drugs.

The result was a publication in The Lancet. At the end, results from students at non-curriculum schools were compared with those of students at Informed Health Choices schools on a 24-question multiple choice test on the 12 concepts (two questions per concept). The results were striking:

The average score for children in the intervention schools was 62·4% (SD 18·8) compared with 43·1% (15·2) in the control schools. The adjusted mean difference (based on the regression analysis) was 20·0% (95% CI 17·3–22·7; p<0·00001) higher in the intervention than in the control group. Appendix 1 shows the distribution of test scores. In the intervention schools, 3967 (69%) of 5753 children had a passing score (≥13 of 24 correct answers), compared with 1186 (27%) of 4430 in the control schools (table 2). The adjusted difference (based on the odds ratio from the logistic regression analysis) was 50% more children who passed (95% CI 44–55; p<0·00001) in the intervention than in the control group.

Noting:

Use of the Informed Health Choices primary school resources had a large effect on the ability of primary school children in Uganda to assess claims about treatment effects. This effect was larger for children with better reading skills, but the intervention was effective for children lacking basic reading skills, as well as for children with basic or advanced reading skills. This effect was achieved even though the learning materials and the tests were in English, which was not the children’s first language. Based on findings from pilot testing both the resources and the test used to measure the outcomes, we were surprised by the size of the effect, which is also large in comparison to other education interventions in primary schools in low-income and middle-income countries,20 and other interventions to teach critical thinking for all ages in high-income countries.11 In addition, the intervention had a positive effect on the children’s intended behaviours and the teachers’ mastery of the key concepts.

Overall, about one-fifth of the children achieved a test score indicating that they had mastered the key concepts (getting at least 20 questions out of the 24 correct), while less than 1% of the children in control schools achieved that high a score. This is quite an effect.

The authors do acknowledge weaknesses in their study. One thing I tend to question is whether the questions on the multiple choice test actually correspond to what the authors say they do. In other words, we don’t know if the children learned how to answer test questions but at the end of the teaching didn’t know how to apply the concepts they learned to real life. We also have no way of knowing what the long term effects of these interventions are and whether they will actually have measurable effects on the health choices made by these participants when they are adults. I also hate to be the pessimist, but I find it disappointing that, even after this intervention, only one-fifth of the children mastered all of the concepts and the average score among those who underwent the teaching still got more than 1/3 of the questions wrong.

As much as there is an emphasis on starting to learn critical thinking at a young age, all is not lost for us old farts:

Separately, the researchers also created a podcast on critical thinking concepts for parents, and tested that approach in another randomized controlled trial, also published in the Lancet. They were successful here as well: Nearly twice as many parents who listened to the podcast series passed a test on their understanding of key health concepts compared with parents in the control group.

Or, in a bit more detail:

We recruited parents between July 21, 2016, and Oct 7, 2016. We randomly assigned 675 parents to the podcast group (n=334) or the public service announcement group (n=341); 561 (83%) participants completed follow-up. The mean score for parents in the podcast group was 67·8% (SD 19·6) compared with 52·4% (17·6) in the control group (adjusted mean difference 15·5%, 95% CI 12·5–18·6; p<0·0001). In the podcast group, 203 (71%) of 288 parents had a predetermined passing score (≥11 of 18 correct answers) compared with 103 (38%) of 273 parents in the control group (adjusted difference 34%, 95% CI 26–41; p<0·0001). No adverse events were reported.

Studies like these, for all their messiness and shortcomings that are unavoidable in carrying out studies like this, give me hope. We human beings have cognitive wiring that leave us prone to making all sorts of incorrect inferences and latching on to all sorts of pseudoscientific beliefs, particularly about health. Critical thinking, while second nature in some areas (such as economic; e.g., when buying a used car), is not natural to most humans, particularly when it comes to health claims, where anecdotes can profoundly mislead and we are very quick to confuse correlation with causation. It can be taught, however. The problem is that there has to be the will and resources to teach it, as well as a large enough core of motivated teachers trained to do it. Then there’s the issue of competing for time among existing subjects that have to be taught. I can’t help but wonder if, rather than teaching critical thinking like this as a module separate from ohter subjects, it would be more effective to find a way to weave training in critical thinking into all subjects, especially science, math, and history, but by no means limited to them and including topics like writing, literature, and other humanities.

However it’s done, the need is acute. 2016 was the year fake news and bullshit appeared to have reached a tipping point. The need to be able to recognize and combat it is more acute than ever, and it’s never too early to start inculcating critical thinking skills into our children, who will need them more than our generation ever did.

By Orac

Orac is the nom de blog of a humble surgeon/scientist who has an ego just big enough to delude himself that someone, somewhere might actually give a rodent's posterior about his copious verbal meanderings, but just barely small enough to admit to himself that few probably will. That surgeon is otherwise known as David Gorski.

That this particular surgeon has chosen his nom de blog based on a rather cranky and arrogant computer shaped like a clear box of blinking lights that he originally encountered when he became a fan of a 35 year old British SF television show whose special effects were renowned for their BBC/Doctor Who-style low budget look, but whose stories nonetheless resulted in some of the best, most innovative science fiction ever televised, should tell you nearly all that you need to know about Orac. (That, and the length of the preceding sentence.)

DISCLAIMER:: The various written meanderings here are the opinions of Orac and Orac alone, written on his own time. They should never be construed as representing the opinions of any other person or entity, especially Orac's cancer center, department of surgery, medical school, or university. Also note that Orac is nonpartisan; he is more than willing to criticize the statements of anyone, regardless of of political leanings, if that anyone advocates pseudoscience or quackery. Finally, medical commentary is not to be construed in any way as medical advice.

To contact Orac: [email protected]

24 replies on “Teaching critical thinking to combat fake news and bullshit: You have to start young”

Hi Orac,

The statements about school curriculum in this post got me thinking about an idea I’ve had for a while. As we all know, the statistics about employments in autistic persons are poor (58% unemployed, ht_tps://www.ncbi.nlm.nih.gov/pubmed/25891885 and ht_tps://www.ncbi.nlm.nih.gov/pubmed/24077909 from that search: (“Unemployment”[Mesh]) AND “Child Development Disorders, Pervasive”[Mesh] ) and it’s not for lack of motivation or education. On the other hand, I do know our governmental agency here in the province will provide a small stipend to both the employee and the employer for part-time work (20 hours per week) and there are other programs for salaried workers depending from where the pool of money come from.

In the case of the 20 hours program, my brother is going for an interview today and tomorrow for a position where he will assemble utensils in bag for restaurants (I’m not sure of the status of the company in question: non-profit or for profit) but in my case, the idea consist of something like Cochrane collaboration coupled with an educational program for (autistic) workers with the result of that work given to the population in the form of scientific publications, layman publications and maybe more as the idea or business grow.

Of course, my comment is a tangent to critical thinking but I have that in mind as a goal too, which is to teach about it, especially in the layman publications area.

Basically, I want to kill many birds (critical thinking, jobs for autistic person, taming the many firehoses of scientific publishing) with one stone.

Alain

I hope they start providing this curriculum to schools here, and in other countries, too. It may not be perfect, but it sounds like it helps.

I would be pleased to have my kids exposed to more critical thinking skills in school (in the context of health choices and in other areas), but I can also imagine the protests that some parents would make, especially if their kids started applying those critical thinking skills to dearly-held political or religious beliefs.

Orac writes,

We human beings have cognitive wiring that leave us prone to making all sorts of incorrect inferences and latching on to all sorts of pseudoscientific beliefs, particularly about health.

MJD says,

Q. Can critical thinking eliminate the placebo effect.

If the answer is NO, integrative medicine is in it’s infancy.

@Heather: Indeed, schools in many US states, most notoriously Texas, are de-emphasizing critical thinking. Because if the kids learn to think critically, they might end up like Molly Ivins:

I believe all Southern liberals come from the same starting point–race. Once you figure out they are lying to you about race, you start to question everything.

Good news–now, how do we get it to go viral? How do we make critical thinking “cool”?

My college student grandson is taking every logic class available and we are having great fun discussing and applying what he’s learning. Grammie is learning some new bits as well! I have to add that he feels he’s pretty much alone in this effort and that he is surrounded by (how to say it?) idiots at his university. They actually have a “pre-chiropractic” major! This is a very well-respected State University.

They actually have a “pre-chiropractic” major!

Which I presume is distinct from a “pre-med” major. Compared to the latter, the “pre-chiropractic” major would presumably have less stringent requirements for organic chemistry, which you don’t really need if you are just going to be manipulating your patients’ spines. But I’m not sure I want to know the details.

I have been fortunate never to be affiliated with a university that had a separate “pre-med” major (there is a nursing program at my present university, but for many in that program it’s a terminal bachelor’s degree). If you want to follow up your undergraduate education with medical school, you have to take an actual major (for obvious reasons biology is the most common choice, but there is no set requirement; humanities, chemistry, physics, and engineering majors have been known to go on to med school) plus whatever additional courses the med schools require that aren’t part of your major or general requirements. I like to think it helps to train potential doctors in critical thinking (at least in the area covered by the major), but it’s not foolproof. Pre-meds are notorious, at least among physics types, for doing minimum work for maximum grade (the ones who actually major in physics are exceptions, because unlike most pre-meds they are actually interested in physics).

Who needs critical thinking courses (and civics, I might add) when we have Internet sites like Infowars and Natural News?

I used to work for a local authority promoting road safety and environmental awareness (yes an odd combination, but that’s local authorities for you). We found that resources which embedded the messages within the existing curriculum were most likely to be adopted by teachers. When a national curriculum was introduced there was a huge opportunity to provide resources that made it easy for teachers to ‘tick the boxes’ they needed to tick. Teachers are under a lot of pressure both in terms of time and money for lesson materials so it is important that resources are of good quality and either provided free or copyright-free especially as you are competing with materials provided by large corporations which can have unhelpful messages alongside the maths english or whatever.. There is a lot to be said for providing resources in a variety of subjects which cover the subject requirements, but which also teach critical thinking.

@JeffM: As Sam Goldwyn said, “If I want your opinion I’ll give it to you.”

In light of my own areas of expertise ( aw… rather than recount, I think you know the drill), SURE, you have to get them young- the foundation of abstract and scientific thought ( think its mathematic roots) lies in pre-formal operational thought so of course the earliest school levels.

re Orac’s last line in the OP
my comment above should also clarify the problems the adults had this year..

When my kids were just kids I would make an effort to get them to question what they saw.
One time stands out in my memory. The kids were in elementary school, and the Fox network (of course) had a credulous special with Uri Geller. When he did his spoon bending I told them to concentrate on watching only his hands. When the host said, “Now we will see…” my daughter cut in with “a man bending a spoon.” They both cracked up. I stopped worrying that they wouldn’t learn to think critically.

Suspected that you would be writing about this article but did not expect it to be the next day. However would like to point out that it also requires “education” to get people to not think. This idea is covered in Suki Kim’s work, “The Art of Essay Writing for the Brainwashed” {https://www.pri.org/stories/2014-12-29/art-essay-writing-brainwashed}, Without You, There Is No Us: My Time with the Sons of North Korea’s Elite {http://www.goodreads.com/book/show/20685373-without-you-there-is-no-us}.

Earlier this year, a colleague sent me the following:

At the University of Washington, biologists were recently deluged with over 20,000 applications for enrollment in a new course titled “Calling Bullshit in the Age of Big Data”. In 20 minutes, the course was entirely booked up. The syllabus and course is free for other colleges and universities to teach, provided acknowledgement of the originators.

The web site for the course offers a bevy of useful resources. As you can see from the course descriptions, one of the subjects to be covered is homeopathy. For a laugh, read the disclaimer on the home page, but don’t mistake the course as a joke.

http://callingbullshit.org/index.html

Is it it critical thinking when you are given the right information? I would rather say that critical thinking is when you understand that what you are told cannot be true.

This study comes out at an interesting time for me. I’ve actually been considering changing careers and going into elementary education for a number of reasons. The chance to introduce critical thinking skills to young people is certainly high on pro side of my pro/con list.

C.C. – I hate to be a downer, but one of the reasons the spousal unit is very glad to be retiring next year (middle school science, civics, geography) is that he does very little actual teaching any more. Most of his classroom time is devoted to preparing the kids to take standardized tests.

Isnt Orac worried that the Teachers Union, or worse, Berkeley kids and faculty, read this piece? After all, you are indicting public school ehjuhmakayshun and gawd knows they might come after you for criticizing their safe spaces

So much for American education.

Panacea, she also thinks too little of teachers. Especially the challenges dealing with a diverse population, and the hormones starting in middle school.

As a former Jr, High Life Science teacher, It can be very hard to even try and teach. I had one class that the average reading percentile was 27. You don’t make much progress with most of the students but work hard to help ones you can.

[…] Rethinking how we teach evaluation Read laterally. Keep it non-political. Switch it up. Lernsystem Informationskompetenz: LIK. Informationskompetenz.org – Wilfried Sühl-Strohmenger. Digitaler Selbstschutz. Teaching critical thinking to combat fake news and bullshit: You have to start young. […]

Comments are closed.

Discover more from RESPECTFUL INSOLENCE

Subscribe now to keep reading and get access to the full archive.

Continue reading