You’ve Been Told It’s Research-Based. But Is It Really?

What teachers should ask when someone says “research shows this works.”

Image created with generative AI

Highlights:

• “Research shows this works.” But what research? Teachers often hear this claim without seeing the evidence behind it.

• Education is especially vulnerable to cycles of trends and innovations.

• Not all research evidence is equally trustworthy—and weaker studies are often overstated.

• This article presents a simple way to examine claims more carefully using Stephen Gorard’s sieve.

• Pro tip at the end: a practical tool teachers can use to investigate “research says…” claims on their own.

  ---

The familiar promise of “research-based”

Most teachers have experienced some version of this moment.

You are sitting in a professional development session or a course in teacher education. A new instructional approach is being introduced. The presenter walks through the strategy, perhaps shows a classroom video, and then at some point says: “Research shows this works.”

In practice, teachers are often asked to accept these claims without seeing the evidence itself. A presenter may say that “research shows” something works and move on, or provide a list of references that appear impressive but turn out, on closer inspection, to be opinion essays or advocacy pieces rather than experimental studies with reliable data. Situations like these are common — and extremely problematic. When the phrase “research shows this works” appears, the conversation often ends before it really begins. If research supports it, the implication is clear: the question has already been settled. But educators should feel comfortable asking this reasonable question:

What kind of research are we referring to?

This is important because in education, the language of evidence is everywhere. Programs are described as evidence-based, strategies are presented as research-informed, and policies are justified with references to the research. Yet in few professions is the word evidence used so frequently and with so little shared agreement about what it actually means (Slavin, 2002).

In medicine, engineering, or aviation, evidence refers to something quite specific: findings derived from systematic investigation, tested against alternatives, and evaluated for reliability. In education, however, the word often means something much broader — and much vaguer. An instructional approach may be described as evidence-based because teachers feel it works, because students appear engaged, because the idea aligns with prevailing beliefs about learning, or because it is presented convincingly in professional development.

None of these things are trivial. Teaching is complex, relational, and deeply contextual. But none of them, on their own, tell us whether an approach reliably improves learning—and that is an essential question we should always be asking as educators.

The cycle of educational innovation

The looseness of the term evidence has big consequences. It allows promising ideas and ineffective ones to circulate side by side, often indistinguishable from each other. Over time, this produces a pattern that many teachers recognize. A new idea arrives with enthusiasm. Schools invest time and energy in professional development and redesign lessons accordingly. For a while the initiative becomes part of everyday practice. Then, a year or two later, attention shifts. The initiative quietly disappears and is replaced by the next innovation.

Rarely does this happen because the earlier idea was carefully evaluated and shown not to work. More often it simply fades from attention (Bryk et al., 2015). Teachers are left with a sense of déjà vu: new initiatives, new terminology, new frameworks—each accompanied by confident claims about evidence.

Examples of this pattern are easy to find. Personality-based frameworks such as True Colors became widely used in schools during the late 1990s and early 2000s. Teachers were encouraged to identify students’ colour profiles using questionnaires and adapt instruction accordingly. Many educators found the framework engaging and affirming. It could promote self-reflection or help people talk about personality differences. What was far less clear, however, was whether such frameworks improved student learning or through what cognitive mechanisms they might do so. Despite their widespread use, they were rarely supported by replicated studies demonstrating instructional impact.

This example is not unusual. Similar cycles have characterized educational reform for decades. Ideas such as learning styles (Pashler et al., 2008), minimally guided discovery learning for novice learners (Kirschner, Sweller, & Clark, 2006), and large-scale intervention programs introduced on the basis of limited or contested evidence have gained widespread traction. Some practices persist even after substantial evidence shows they do not produce the learning outcomes they promise (Coffield et al., 2004; De Bruyckere, Kirschner, & Hulshof, 2015). These ideas spread not necessarily because they work, but because they resonate with compelling beliefs about learning and teaching. Once embedded in professional culture, they are rarely subjected to sustained evaluation. They tend to fade—if they fade at all—not through disconfirmation but through replacement.

Paul Kirschner, Carl Hendrick, and Jim Heal describe this pattern as the innovation illusion in their book Instructional Illusions: the assumption that innovation itself leads to improvement, that new automatically means better. In education, novelty is often mistaken for progress. The result is a profession that often reinvents itself rather than steadily accumulating knowledge about what works. Instructional time is far too valuable for cycles of reinvention that often fail to produce durable learning for students. It is worth asking why does this happen so frequently in education.

Ongoing resistance to evidence

Educational researcher Douglas Carnine offered one of the clearest explanations of why this cycle persists. In his essay Why Education Experts Resist Effective Practices, Carnine (2000) argued that education often struggles to operate as a science-based profession. Instructional approaches shown to be effective in large-scale studies have sometimes been rejected not because they failed, but because they conflicted with dominant beliefs about how teaching ought to look.

We explored this tension in more detail in an earlier post, Beyond Belief: Reframing Teaching as a Science-Based Profession, where we examined Carnine’s argument that education often operates less like fields such as medicine or engineering and more like a belief-driven profession. When instructional ideas spread primarily because they align with prevailing beliefs about learners or classrooms, evidence struggles to exert corrective influence—even when it is strong.

Reading instruction provides a striking example. Decades of research have examined how children learn to read and which instructional approaches are most effective. Yet the central challenge has rarely been the absence of research.

As Jeanne Chall observed in her book The Academic Achievement Challenge: What Really Works in the Classroom?:

“The problem, in beginning reading, is not the research. The results of the research on beginning reading have been the same since the 1920s. The problem is getting people to accept the results of this research.”

Her observation captures a recurring tension in education: evidence may exist, but it does not automatically shape practice.

Cognitive scientists Paula Stanovich and Keith Stanovich help explain why this tension is so persistent. Research on human reasoning shows that people naturally prioritize vivid examples, personal experience, and ideas that already feel intuitively right (Stanovich, 2003; Stanovich & Stanovich, 2003). In educational contexts this tendency becomes problematic when experience and intuition are treated as sufficient evidence in their own right, rather than being checked against systematic research.

Personal experience is valuable in teaching. Experienced teachers develop deep professional knowledge about students and classrooms. But as Stanovich and Stanovich note, experience alone is not sufficient for judging whether an instructional strategy improves learning. Professions differ not in whether they are vulnerable to these biases—they all are—but in how well they protect themselves against them. In science-based professions, personal experience is valued — but it is checked against systematic research. As Stanovich and Stanovich put it:

Drawing upon personal experience is necessary and desirable in a veteran teacher, but it isn’t sufficient for making critical judgments about the effectiveness of an instructional strategy.

When experience and intuition dominate, appealing ideas can circulate for years without being carefully tested. Over time the word evidence begins to lose its meaning.

Mirjam Neelen and Paul Kirschner describe this problem using the distinction between truth and truthiness. The term truthiness, originally coined by comedian Stephen Colbert, refers to claims that feel true because they align with intuition or belief — not because they are supported by strong evidence. Education is full of ideas that carry the appearance of truthiness. They are persuasive because they feel intuitive, produce engaging classroom activities, generate enthusiastic student responses, and align with appealing narratives about innovation. But plausibility is not the same as reliability. And without clear standards for evaluating research, it becomes difficult to distinguish between the two.

Not all research evidence is equal

Another challenge in education is that studies of very different quality are often treated as if they carry the same weight.

Researcher Stephen Gorard has argued that this is one reason education struggles to accumulate reliable knowledge. When weak and strong studies are treated as equivalent, dramatic but fragile findings can overshadow more modest yet far more reliable evidence (Gorard, 2014; Gorard, 2024).

One of the most important observations in research methodology helps explain why this happens: weaker studies often end up producing the strongest claims. This does not mean that weak studies themselves are a problem. Early research in any field often begins with small or exploratory studies. Nor is it unusual for studies to report findings that later turn out to be overstated. The difficulty arises in what happens after those studies appear. If a study aligns with an idea people already find appealing, it can quickly be treated as stronger evidence than it actually is. A study based on a small sample, a weak comparison, or indirect measures of learning may still be cited repeatedly as proof that a particular approach “works.” Over time, the limitations of the original study fade from view while the claim itself becomes amplified.

In other words, the issue is not that weak studies exist. The issue is that they are often cherry-picked and presented as if they provide decisive evidence. When this happens, findings circulate widely—especially when they align with prevailing beliefs or current educational trends. The result is a familiar cycle: enthusiasm, adoption, and eventually disappointment when the promised impact does not materialize.

So what can teachers do?

Recognizing these problems is important, but it also raises a practical question for teachers. When someone says “research shows this works,” how can teachers begin to evaluate that claim more carefully? What questions should they ask before placing confidence in the evidence being presented?

To help address this problem, Gorard proposed a practical way of thinking about research trustworthiness, sometimes referred to as Gorard’s sieve (see visual above). The idea is straightforward: before asking whether research should guide practice, we should first ask whether the research itself is trustworthy. Four simple questions provide an initial filter:

• Does the study design actually allow researchers to answer the question they are asking?

• How large are the groups being compared?

• How much data is missing?

• And how is learning measured?

These questions do not require teachers to become research methodologists. But they create an opportunity to pause and ask a more disciplined question:

How secure is this evidence, really?

Strengthening professional judgment

The solution is not to abandon research or professional judgment. Quite the opposite. Effective teaching draws on multiple forms of knowledge: knowledge of subject matter, learners, instruction, and tools. Evidence matters because it connects these forms of knowledge to action. It helps teachers judge which instructional approaches are likely to support learning, under what conditions they may work, and why.

Clarifying what counts as evidence is therefore not about constraining teachers’ judgment. It is about strengthening it. Because in the end, the question that matters most is a simple one:

Does this approach actually help students learn?

Evidence, when used carefully, is what allows the profession to answer that question with increasing confidence.

Pro Tip: Teachers can also use generative AI as a tool (Perplexity, Claude or ChatGPT) to help examine educational claims more carefully. To support this, we’ve created a short guide with a prompt you can use to explore common “research says…” claims and what the research literature actually shows. You can access the guide here: A Teacher’s Prompt for Evaluating “Research Says…” Claims

References:

Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America’s schools can get better at getting better. Harvard Education Press.

Carnine, D. (2000). Why education experts resist effective practices (and what it would take to make education more like medicine). Thomas B. Fordham Foundation.

Chall, J. S. (2000). The academic achievement challenge: What really works in the classroom? Guilford Press.

Coffield, F., Moseley, D., Hall, E., & Ecclestone, K. (2004). Learning styles and pedagogy in post-16 learning: A systematic and critical review. Learning and Skills Research Centre.

De Bruyckere, P., Kirschner, P. A., & Hulshof, C. D. (2015). Urban myths about learning and education. Academic Press.

Gorard, S. (2014). A proposal for judging the trustworthiness of research findings. Radical Statistics, 110, 47–59.

Gorard, S. (2024). Judging the relative trustworthiness of research results: How to do it and why it matters. Review of Education, 12(1), e3448. https://doi.org/10.1002/rev3.3448

Kirschner, P. A., Hendrick, C., & Heal, J. (2025). Instructional illusions: Why teaching myths persist—and what we can do about them. Hachette Learning.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1

Lilienfeld, S. O., Lynn, S. J., Namy, L., Woolf, N., Jamieson, G., Marks, A., & Slaughter, V. (2014). Psychology: From inquiry to understanding (2nd ed.). Pearson.

Neelen, M., & Kirschner, P. A. (2020, July 7). Truth or truthiness? Analysing a VR study using Gorard’s sieve. 3-Star Learning Experiences. https://3starlearningexperiences.wordpress.com/2020/07/07/truth-or-truthiness-analysing-a-vr-study-using-gorards-sieve/

Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105–119. https://doi.org/10.1111/j.1539-6053.2009.01038.x

Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15–21. https://doi.org/10.3102/0013189X031007015

Stanovich, K. E. (2003). How to think straight about psychology (7th ed.). Pearson.

Stanovich, P. J., & Stanovich, K. E. (2003). Using research and reason in education: How teachers can use scientifically based research to make curricular and instructional decisions. National Institute for Literacy.