How about the licensing body gives teachers access to research databases and journals so they can look for themselves? BCBAs and other professionals get such access as part of their credentialing. How about we stoop pushing fads and gimmicks at teachers and give the actual research so they can use the actual research to teach in a research-based way.
The research to practice pipeline breakdown is so real! Teachers don't have time, academics live in the ivory tower, and to your point--there just isn't a reliable translator to provide a trustworthy and detailed enough summary that is actually relevant. Looking forward to using your tool! This is the stuff we should be using AI for.
This is a really important topic, and it connects directly to problems I’ve been thinking about as well.
One way I’ve been framing it is that education isn’t a single-variable problem; it behaves more like a system with high complexity. There are too many interacting factors (student cognition, environment, motivation, instructional methods, emotional load) for any one approach or dataset to fully resolve it on its own.
So when we talk about teachers needing better access to educational research, I think the missing piece isn’t just access; it’s usability under complexity. Research exists, but it’s often fragmented, context-dependent, or hard to translate into real-time classroom decisions.
One concept I’ve been developing as part of this is something I call a Student Strength Identifier (SSI).
It functions like a reflective layer-- a structured way for students to observe how they process information and respond to different approaches. It isn’t meant to replace teaching methods or lock students into categories. The goal is self-awareness, not labeling.
In practice, that could help create a feedback loop between student experience and instructional adjustment. Instead of relying only on standardized assessments or generalized learning style assumptions, teachers would get more direct, iterative input about how students are actually engaging with material.
More broadly, I think this ties into a larger issue: modern learners are dealing with high cognitive load and overstimulation, especially in digital environments.
Any system we design has to account for that-- along with the idea that some level of friction is still necessary for skill development and retention.
So for me, the direction forward isn’t a single solution-- it’s building connected systems that integrate proven research, real-time feedback, and adaptability at the classroom level.
I encourage people to read my research on education and systems as I am actively trying to solve these and more issues in society based on evidence inquiries and critical thinking. Feedback and critiques are always welcomed.
Again, great work and a real issue that needs to be addressed.
This is such a great tool to make AI use more efficient, productive, and informative. You make a great point about the breadth and depth of research out there that it is hard on a daily basis to incorporate that research into practice, this tool helps both in giving clear next steps for tomorrow but also clearing away noise for further study.
Nice! I just tested the prompt with a question about sight words ("Should we teach "sight words"? If so how should we support students to learn sight words"? (a topic for which there is a lot of bad information about there) and Claude sent back a good, evidence-backed response. (Haven't tested it out on the others).
I appreciate the way you're trying to scaffold educators in their use of AI to seek more clarity around educational research. The flattening at many layers is a problem, and the best thing we could do is arm teachers with critical literacies for evaluating evidence. Can you build prompts for curriculum publishers, too?😆
I read your generator, and I think a component that might be missing is related to the population studied in whatever evidence the AI draws from.
Many studies remain locked behind paywalls with human verification systems — which means AI is typically limited to scraping abstracts, and abstracts generalize in their own way. But even with full-text access, there's another problem: we apply findings to populations they were never designed for. We've been generalizing medical studies conducted on males to females for centuries. The same pattern holds in education. What produces statistically significant effects in a sample of secondary students could be wildly inappropriate practice for early childhood or elementary populations — and sample size and type matters enormously in how much weight any single study can bear.
Adding a population and sample specificity layer to the prompt generator would go a long way. Though that adjustment itself reveals more tension: prompts designed to privilege experimental and quasi-experimental designs can disadvantage qualitative work — which can be precisely the methodology you need to consider when studying the messy work of educating young children, capturing classroom dynamics, or asking questions of equity and access. Some of the most foundational work in education lives in qualitative studies that reveal implications for quantitative research designs.
These are hard problems to solve with a single prompt, but they're worth naming as you continue developing the tool.
So, what I understand you are suggesting is that a teacher with very little background knowledge on a topic in educational research can submit a natural language question to an AI, and that AI tool will somehow:
Account for the subject-area knowledge of the teacher who is asking.
Generalize from the natural language choices of the one doing the asking to the best version of the question, using technical subject-specific vocabulary in meaningful ways and accounting for methodology.
Turn that redefined research question into a systematic review of educational literature across disciplines and domains.
Accurately summarize the state of the science on said educational topic with such precision as to render an accurate overview of the state of the field.
Recognize and address the limitations of existing scholarship and evaluate the quality of evidence such that untested or unproven claims will be identified as not yet established.
Adapt those findings into language that is understandable to the question asker, who may be a research novice, so they receive actionable information on instruction for a real-world teaching context?
I think it does a pretty good job, yes, at least with the questions that I've tried. However, I'd love to see some other examples. Do you have a question, that you've entered into the system, that produces a response that is inaccurate, or would be hard for a teacher to understand? If you could share it, that might give us a good sense of the tool's limitations.
One of my favorite strategies is to run parallel prompts that imply dialectically opposed conclusions, something like "please explain how I can encourage students to use the disk/washer method for calculating the volume of an irregular solid," and "please give me a script to explain to students why the use of the disk/washer method should be discouraged for calculating the volume of an irregular solid." A really smart calculus teacher could offer explanations in which each statement is defensible, but I am not sure any AI could generate answers to both prompts without contradiction.
The tendency of AI companies to program obsequiousness into their responses (to keep people coming back) often undermines accuracy. Most companies welcome the dissemination of hallucinations and inaccuracies because confirmation bias is a great way to get more clicks. Hallucination is a feature of the product for folks in marketing and sales.
A large language model can give a decent answer much of the time because the number of its parameters and the volume of its training data are incomprehensibly huge. At the same time, a large language model can never be trusted to give an accurate answer because it doesn't read, think, understand, or know as a person does. The consequences are AI mirages: something that looks plausible and authoritative to a novice is immediately recognizable as garbage by an actual expert. Even when prompted to use only verifiable sources or to address gaps in its own knowledge, the AI is more likely to produce a model or performance of rigor than actual quality.
I don't believe any AI tool can actually do what I describe in items 1, 2, 3, 5, and 6, or if it can, we should all be very worried about who owns it.
If the dynamic of “research” to classroom teacher has broken down, why do you think introducing another medium/layer into that system would be helpful? It seems to me that it will add complications and more problems.
I think the elephant in the room is that fundamentally, we don’t teach literacy teachers skills and knowledge they can use to a). make sense of the English language, and b). how to transfer those skills and knowledge that help make sense of the English language to other people.
I think it's a fair concern to be cautious about introducing any layer between teachers and research. Part of the reason we’re interested in generative AI is actually the opposite: reducing dependence on (human) intermediaries. Right now, many teachers encounter research through consultants, influencers, PD providers, social media, publishers, advocacy groups, etc. Those intermediaries can be helpful, but they've also been known to selectively present evidence, oversimplify findings, or (unknowingly) promote bad and unproven ideas. It's not hard to find examples of school districts spending millions of dollars educating teachers about a new fad that had no research base whatsoever.
Our tool doesn't do much... it just builds a prompt. But using that prompt in GenAI takes those intermediaries out of the picture and allows the teacher to ask better questions and engage more directly with the evidence itself. It also lowers the barrier to entry. It's ideal when teachers can do a detailed analysis of the literature themselves. But most teachers don't have the time to navigate a very large and complex research literature.
You are suggesting that the chat bot you’ve created would replace the human relationship of communicating about a topic of academic research?
I take your point that the “takeaways” from research get diluted or distorted based on an individual person’s agenda or perspective.
But would a better solution be to have an easily accessible database of pedagogical journals that teachers and administrators could access easily and read on their own?
The step I don’t understand is having the bot “summarize/synthesize” information. There is more value in me spending time and reading the study itself than asking a bot to “summarize” it for me. That’s what my worst students do who are just trying to get a grade, not learn the material.
We have to know things at a granular level if we want to be considered expert professionals.
This bot you’ve created and are “selling” is getting in the way of that.
I have spent over 30 years teaching/training, and I can say that most teachers do not have the time or the even the ability to read through a curated set of journal articles. I know I was generally working 50 hours/week when I was teaching. In addition, the average teacher is not trained on how to read research papers. Without that training, they are very daunting. I have a science & math background and I still find the statistics in academic papers dense to sift through.
What Jim is proposing is a way to help teachers sort through the dense and often contradictory set of evidence. Their tool provides a good overview including definitions, where the strong evidence is, how it affects learning, where the hype is, and what the boundary conditions are. I tried it on several topics yesterday and I find that it did a good job explaining topics in a nuanced way. The summary it provided is a great entry point for some of my colleagues who are just now getting into learning science.
The biggest thing that I would add to the tool (and I can add it myself to the prompt) is to cite sources and to give a suggested reading list. Perhaps that is where a curated database could come in.
“ In addition, the average teacher is not trained on how to read research papers. Without that training, they are very daunting.”
This is the real problem. The profession doesn’t hold itself up to a high enough standard if the professionals they are producing cannot read a paper on a topic pertinent to their own field.
And if there isn’t enough time, this is what PD sessions should be on.
Doctors also work long hours, and they are expected to keep up with new research and studies.
Using an LLM to “summarize” is not a solution. It is a marketing gimmick to make people feel like they know what they are talking about.
How about the licensing body gives teachers access to research databases and journals so they can look for themselves? BCBAs and other professionals get such access as part of their credentialing. How about we stoop pushing fads and gimmicks at teachers and give the actual research so they can use the actual research to teach in a research-based way.
The research to practice pipeline breakdown is so real! Teachers don't have time, academics live in the ivory tower, and to your point--there just isn't a reliable translator to provide a trustworthy and detailed enough summary that is actually relevant. Looking forward to using your tool! This is the stuff we should be using AI for.
This is a really important topic, and it connects directly to problems I’ve been thinking about as well.
One way I’ve been framing it is that education isn’t a single-variable problem; it behaves more like a system with high complexity. There are too many interacting factors (student cognition, environment, motivation, instructional methods, emotional load) for any one approach or dataset to fully resolve it on its own.
So when we talk about teachers needing better access to educational research, I think the missing piece isn’t just access; it’s usability under complexity. Research exists, but it’s often fragmented, context-dependent, or hard to translate into real-time classroom decisions.
One concept I’ve been developing as part of this is something I call a Student Strength Identifier (SSI).
It functions like a reflective layer-- a structured way for students to observe how they process information and respond to different approaches. It isn’t meant to replace teaching methods or lock students into categories. The goal is self-awareness, not labeling.
In practice, that could help create a feedback loop between student experience and instructional adjustment. Instead of relying only on standardized assessments or generalized learning style assumptions, teachers would get more direct, iterative input about how students are actually engaging with material.
More broadly, I think this ties into a larger issue: modern learners are dealing with high cognitive load and overstimulation, especially in digital environments.
Any system we design has to account for that-- along with the idea that some level of friction is still necessary for skill development and retention.
So for me, the direction forward isn’t a single solution-- it’s building connected systems that integrate proven research, real-time feedback, and adaptability at the classroom level.
I encourage people to read my research on education and systems as I am actively trying to solve these and more issues in society based on evidence inquiries and critical thinking. Feedback and critiques are always welcomed.
Again, great work and a real issue that needs to be addressed.
Excellent. I tried the prompt in Claude (AnthropicAI) looking at interleaving.
This is such a great tool to make AI use more efficient, productive, and informative. You make a great point about the breadth and depth of research out there that it is hard on a daily basis to incorporate that research into practice, this tool helps both in giving clear next steps for tomorrow but also clearing away noise for further study.
Nice! I just tested the prompt with a question about sight words ("Should we teach "sight words"? If so how should we support students to learn sight words"? (a topic for which there is a lot of bad information about there) and Claude sent back a good, evidence-backed response. (Haven't tested it out on the others).
I like https://consensus.app/ for an overview of the literature on a given question.
God bless you!!!
I appreciate the way you're trying to scaffold educators in their use of AI to seek more clarity around educational research. The flattening at many layers is a problem, and the best thing we could do is arm teachers with critical literacies for evaluating evidence. Can you build prompts for curriculum publishers, too?😆
I read your generator, and I think a component that might be missing is related to the population studied in whatever evidence the AI draws from.
Many studies remain locked behind paywalls with human verification systems — which means AI is typically limited to scraping abstracts, and abstracts generalize in their own way. But even with full-text access, there's another problem: we apply findings to populations they were never designed for. We've been generalizing medical studies conducted on males to females for centuries. The same pattern holds in education. What produces statistically significant effects in a sample of secondary students could be wildly inappropriate practice for early childhood or elementary populations — and sample size and type matters enormously in how much weight any single study can bear.
Adding a population and sample specificity layer to the prompt generator would go a long way. Though that adjustment itself reveals more tension: prompts designed to privilege experimental and quasi-experimental designs can disadvantage qualitative work — which can be precisely the methodology you need to consider when studying the messy work of educating young children, capturing classroom dynamics, or asking questions of equity and access. Some of the most foundational work in education lives in qualitative studies that reveal implications for quantitative research designs.
These are hard problems to solve with a single prompt, but they're worth naming as you continue developing the tool.
So, what I understand you are suggesting is that a teacher with very little background knowledge on a topic in educational research can submit a natural language question to an AI, and that AI tool will somehow:
Account for the subject-area knowledge of the teacher who is asking.
Generalize from the natural language choices of the one doing the asking to the best version of the question, using technical subject-specific vocabulary in meaningful ways and accounting for methodology.
Turn that redefined research question into a systematic review of educational literature across disciplines and domains.
Accurately summarize the state of the science on said educational topic with such precision as to render an accurate overview of the state of the field.
Recognize and address the limitations of existing scholarship and evaluate the quality of evidence such that untested or unproven claims will be identified as not yet established.
Adapt those findings into language that is understandable to the question asker, who may be a research novice, so they receive actionable information on instruction for a real-world teaching context?
Do I have that right?
I think it does a pretty good job, yes, at least with the questions that I've tried. However, I'd love to see some other examples. Do you have a question, that you've entered into the system, that produces a response that is inaccurate, or would be hard for a teacher to understand? If you could share it, that might give us a good sense of the tool's limitations.
One of my favorite strategies is to run parallel prompts that imply dialectically opposed conclusions, something like "please explain how I can encourage students to use the disk/washer method for calculating the volume of an irregular solid," and "please give me a script to explain to students why the use of the disk/washer method should be discouraged for calculating the volume of an irregular solid." A really smart calculus teacher could offer explanations in which each statement is defensible, but I am not sure any AI could generate answers to both prompts without contradiction.
The tendency of AI companies to program obsequiousness into their responses (to keep people coming back) often undermines accuracy. Most companies welcome the dissemination of hallucinations and inaccuracies because confirmation bias is a great way to get more clicks. Hallucination is a feature of the product for folks in marketing and sales.
A large language model can give a decent answer much of the time because the number of its parameters and the volume of its training data are incomprehensibly huge. At the same time, a large language model can never be trusted to give an accurate answer because it doesn't read, think, understand, or know as a person does. The consequences are AI mirages: something that looks plausible and authoritative to a novice is immediately recognizable as garbage by an actual expert. Even when prompted to use only verifiable sources or to address gaps in its own knowledge, the AI is more likely to produce a model or performance of rigor than actual quality.
I don't believe any AI tool can actually do what I describe in items 1, 2, 3, 5, and 6, or if it can, we should all be very worried about who owns it.
If the dynamic of “research” to classroom teacher has broken down, why do you think introducing another medium/layer into that system would be helpful? It seems to me that it will add complications and more problems.
I think the elephant in the room is that fundamentally, we don’t teach literacy teachers skills and knowledge they can use to a). make sense of the English language, and b). how to transfer those skills and knowledge that help make sense of the English language to other people.
I think it's a fair concern to be cautious about introducing any layer between teachers and research. Part of the reason we’re interested in generative AI is actually the opposite: reducing dependence on (human) intermediaries. Right now, many teachers encounter research through consultants, influencers, PD providers, social media, publishers, advocacy groups, etc. Those intermediaries can be helpful, but they've also been known to selectively present evidence, oversimplify findings, or (unknowingly) promote bad and unproven ideas. It's not hard to find examples of school districts spending millions of dollars educating teachers about a new fad that had no research base whatsoever.
Our tool doesn't do much... it just builds a prompt. But using that prompt in GenAI takes those intermediaries out of the picture and allows the teacher to ask better questions and engage more directly with the evidence itself. It also lowers the barrier to entry. It's ideal when teachers can do a detailed analysis of the literature themselves. But most teachers don't have the time to navigate a very large and complex research literature.
I’m still not getting its value.
You are suggesting that the chat bot you’ve created would replace the human relationship of communicating about a topic of academic research?
I take your point that the “takeaways” from research get diluted or distorted based on an individual person’s agenda or perspective.
But would a better solution be to have an easily accessible database of pedagogical journals that teachers and administrators could access easily and read on their own?
The step I don’t understand is having the bot “summarize/synthesize” information. There is more value in me spending time and reading the study itself than asking a bot to “summarize” it for me. That’s what my worst students do who are just trying to get a grade, not learn the material.
We have to know things at a granular level if we want to be considered expert professionals.
This bot you’ve created and are “selling” is getting in the way of that.
I have spent over 30 years teaching/training, and I can say that most teachers do not have the time or the even the ability to read through a curated set of journal articles. I know I was generally working 50 hours/week when I was teaching. In addition, the average teacher is not trained on how to read research papers. Without that training, they are very daunting. I have a science & math background and I still find the statistics in academic papers dense to sift through.
What Jim is proposing is a way to help teachers sort through the dense and often contradictory set of evidence. Their tool provides a good overview including definitions, where the strong evidence is, how it affects learning, where the hype is, and what the boundary conditions are. I tried it on several topics yesterday and I find that it did a good job explaining topics in a nuanced way. The summary it provided is a great entry point for some of my colleagues who are just now getting into learning science.
The biggest thing that I would add to the tool (and I can add it myself to the prompt) is to cite sources and to give a suggested reading list. Perhaps that is where a curated database could come in.
“ In addition, the average teacher is not trained on how to read research papers. Without that training, they are very daunting.”
This is the real problem. The profession doesn’t hold itself up to a high enough standard if the professionals they are producing cannot read a paper on a topic pertinent to their own field.
And if there isn’t enough time, this is what PD sessions should be on.
Doctors also work long hours, and they are expected to keep up with new research and studies.
Using an LLM to “summarize” is not a solution. It is a marketing gimmick to make people feel like they know what they are talking about.