There are moments in academic life when you come across a paper and wonder how nobody had written it before. Pat Thomsonhas recently published one of them. Her article, in the Journal of Education Policy, develops a five-level Policy Bullshit Index for assessing claims made in the name of evidence. Not weak evidence. Not limited evidence. Not the more refined academic formulation that “the conclusions may extend beyond what can reasonably be inferred from the available data”.
Bullshit. In a peer-reviewed journal. Finally.
Anyone who has spent enough time around education policy will recognise the problem immediately. A report is released. It contains references. There are graphs, a systematic review or two, and preferably a foreword from somebody important. Then comes the line: the evidence shows. From there, a complicated literature is converted into a remarkably uncomplicated policy recommendation. Schools should do this. Teachers should stop doing that. Universities need to change their courses. The research is settled. Best practice has been identified. What works is now known. Sometimes the evidence really is that clear, but more often the confidence exceeds what the research can support.
This is not an argument against evidence. I have made that point often enough that I no longer need to defend it. Explicit instruction works. Cognitive load theory tells us important things about learning. Systematic phonics has a considerable evidence base. Some teaching practices are demonstrably more effective than others, particularly for particular learners, purposes and stages of learning. I concede all of this. The problem starts when a useful body of evidence becomes a universal claim; a universal claim then becomes a policy prescription, and that prescription is repeatedly defended by declaring it “evidence-based”.
Thomson’s contribution is to give this move a name and, more importantly, a way of examining it. Her definition of policy bullshit is not simply “a policy I disagree with”, nor is bullshit meant to be synonymous with dishonesty. A claim can be factually accurate and still earn its place on the index. The issue is whether the evidential authority being claimed is actually supported by the foundational research. Her index looks at three areas: where the evidence comes from and how it was produced, the quality of the research itself, and the quality of the argument built from it. In other words: who produced this, how good is the evidence, and is the conclusion supported by the data?
None of those questions is revolutionary. They are the kinds of questions academics ask when reviewing journal articles, examining doctoral theses or assessing research. Thomson is not claiming to have discovered a new form of methodological appraisal. What she has done is package the familiar standards of academic review into a form that can compete rhetorically with the certainty of policy.
Academics are exceptionally good at saying that things are complex. Give us a questionable policy claim, and we can produce 10,000 words explaining the limitations of the sampling frame, the inadequacy of the comparator, the failure to consider rival explanations and the inappropriate generalisation from six carefully selected cases. However, by the time that is published, the minister has announced the policy, the department has created the implementation team, and somebody is already making money on the professional learning program for teachers. But the problem isn’t only the time lag.
Scholarly critique has to operate in a world where ministers need announcements, journalists need headlines, and policy organisations need recommendations. Nuance is important, but nuance that nobody notices has limited capacity to challenge overclaiming. “The evidence base for this proposal is methodologically limited, and the conclusions are disproportionate to the data” may be precise academic jargon; however, Level Three: High Bullshit lands a bit more forcefully. Thomson is quite explicit that this rhetorical force is the point.
She demonstrates the applicability of the index by applying it to the Grattan Institute’s Spreading Success report, which advocates trials of multi-school organisations in Australia based partly on examples from England and New York. Thomson does not say that she dislikes the proposal. She follows the evidence, examines the cases, considers what has been omitted, and asks whether the research design supports the recommendations. I’ll let you decide how well it does that based on how the report scored on the index: Level Three: High Bullshit. Pretty clear, isn’t it? Although she offers plenty of elaborations, I think we all get the gist.
I have no particular interest in multi-school organisations; that is not the interesting part of the paper. For me, the important distinction is between saying the proposal is wrong and saying the evidence presented does not justify the confidence with which the proposal is being promoted. Those are very different claims, and education policy routinely collapses them. By labelling it Level Three: High Bullshit, we get to skip the academic jargon – the point lands with everyone.
We have created an environment where challenging an evidence claim is treated like challenging the evidence itself. Question an interpretation of cognitive load theory, and you are automatically deemed anti-cognitive science. Question whether explicit instruction should become the organising pedagogy of an entire system and you are somehow pro-chaos. Question whether the evidence actually supports the policy, and suddenly you are accused of ignoring the evidence.
Researchers decide which questions to ask, what to measure, which studies to include, which outcomes matter and how far findings can reasonably extend beyond the circumstances in which they were produced. Policy organisations then make another series of choices about which research to cite, which findings to emphasise, which qualifications to retain and which inconvenient complexities to disappear. None of this is inherently corrupt or wrong. It is how knowledge works. The bullshit arrives when those choices vanish, and the resulting policy is presented as though the evidence could only possibly have led to one conclusion.
Education may be especially vulnerable because policy and research operate under different incentives. Research, at its best, rewards doubt. We qualify conclusions, discuss limitations and remain open to alternative explanations. Policy eventually has to do something. Governments cannot issue a policy statement saying the evidence is interesting but heterogeneous and that further investigation of contextual moderators would be desirable. That doesn’t fit on a bumper sticker. They either make decisions, or decide not to. The problem is pretending the uncertainty disappeared before they decided.
I would have more confidence in an education policy that said: the evidence is mixed, but on balance we think this is the best option and here is why than one insisting that the evidence is clear when it plainly is not. One invites scrutiny, allows for nuance and accepts responsibility for the judgement. The other borrows the authority of science to make the judgement appear inevitable.
That distinction matters because “evidence-based” has become an effective way to shut down debate. Once a policy is positioned as the evidence-based option, alternatives are no longer simply alternatives. They become unevidenced, ideological, or anti-science. The disagreement has been reframed before it begins. One side owns “the evidence”; everyone else apparently owns opinions.
This is where Thomson’s terminology earns its keep. Calling something bullshit interrupts that rhetorical exercise. It asks whether the authority being claimed has actually been earned. Have all the relevant studies been considered, including the inconvenient ones? Are the methods capable of answering the question being asked? Do the sources actually support the claims attributed to them? Is the recommendation proportionate to the evidence?
The most important aspect of the Policy Bullshit Index is that it does not give critics permission to simply label policies they dislike as bullshit. Thomson’s index requires work. Applying it means analysing the document closely, checking sources, examining the wider literature and evaluating both research quality and argumentation. The incendiary name comes with a useful limitation: if you are going to call bullshit, you’d better be able to show your work. And that may be exactly what education policy needs.
I have written before about the arrogance of certainty in education. This paper gets at something else. The problem is not simply that we are too certain. It is that certainty now has a vocabulary recognisable to every educator: evidence-based, what works, best practice, the science of learning. These phrases signal serious engagement with research. But they also function as a rhetorical shield, making a contested interpretation sound like an empirical fact.
What evidence? How strong is it? What has been excluded? What alternative explanations exist? Does the research demonstrate what the policy document says it demonstrates? And, most importantly, does the recommendation go further than the evidence can reasonably justify?
Those are ordinary academic questions. Thomson’s achievement – and the Journal of Education Policy deserves some credit for publishing the paper without sanitising the terminology – is recognising that we needed a more provocative label if we expect anyone outside academia to pay attention.
Education does not need less evidence. It needs considerably more honesty about what evidence can and cannot tell us. Governments, think tanks, advocacy organisations and researchers should all be able to make arguments, recommend policies and exercise judgement. But if they want to borrow the authority of research, they have an obligation to represent that research honestly.
If the evidence is strong, show us. If it is contested, say so. If judgement is doing some or most of the work, own it. And if the conclusion only loosely reflects the research it claims to represent, we now have a peer-reviewed, citable term for it: Bullshit.
