‘Technology does not work in the same way or to the same effect in all classrooms and with all students.’ In a 3-part series on technology in education, Dr Ralph Saubern, Deputy CEO of the Australian Council for Educational Research, explores whether the integration of technology in schools has led to improved outcomes for learners. In part 1, he looked at a developmental approach to building teacher knowledge and in part 2, he questioned how schools and teachers can choose the right EdTech in the first place. In this final instalment, Dr Saubern addresses the challenges – and opportunities – in evaluating the impact of EdTech on learning outcomes.
One of the most interesting parts of my doctoral research was reviewing the literature on the effectiveness of educational technology. As I worked through the research, trying to find clear evidence that technology was helping schools and teachers improve student learning outcomes, a surprising picture began to emerge. While some studies showed a positive impact in specific contexts, there was little strong evidence that technology was significantly improving learning outcomes overall. It wasn’t what I expected.
In an appropriately named paper, Virtually no effect?, Falck, Mang and Woessmann systematically reviewed studies that considered the effect of classroom computers on student achievement (Falck et al., 2018). Other large-scale studies and meta-analyses reached similar conclusions. At best, the effect of technology is modest and context-dependent; in some cases, the impact is negative. A recent study covering 23 countries found no link between technology adoption and improved efficiency in education systems (Mergoni et al., 2023).
This is challenging. We have a good, evidence-based understanding of what works in education, and on the face of it, educational technologies have many of the features and qualities that should support and magnify those approaches: feedback, personalised learning, formative assessment, access and equity, and more. So why can’t we see a stronger effect on learning outcomes from EdTech?
In his Visible Learning (n.d.) work, John Hattie conducted meta-analyses of educational influences and interventions, combining many smaller studies into a single, meta-analysis that gives us more confidence to make generalisations. His analysis of EdTech-related interventions found varied results.
‘Online and digital tools’ showed an overall effect size of 0.38, just under the 0.40 threshold Hattie suggests for identifying worthwhile interventions. ‘Intelligent tutoring systems’ performed better at 0.52. ‘Gaming in language’ showed a large effect size of 1.04, though with low confidence due to the small number of studies and participating students. At the other end of the scale, ‘one-to-one laptops’ showed a weak positive effect size of 0.16, and the ‘presence of mobile phones’ was associated with a negative effect size of -0.24.
Evaluating EdTech: A moving target
So, as is often the case in educational research, the answer to whether EdTech works is: it depends. This does not help teachers and schools in making day-to-day decisions. To complicate matters, EdTech is constantly evolving. New features, frequent updates, and emerging technologies make it difficult to evaluate tools before they change again.
Some organisations have attempted to address this by developing evaluation systems that are more practical for educators. For example, the global nonprofit Digital Promise has created a product certification program for EdTech tools. Common Sense Education has a star rating system. In combination with research summaries and evidence guides from organisations such as Evidence for Learning and the Australian Education Research Organisation, these resources can help schools make more informed decisions.
Where’s the teacher?
In my PhD research, I approached the problem from a different angle: I focused on the role of the teacher. Research shows that technology is not always used in ways that maximise its potential, and that classroom context matters. Technology does not work in the same way or have the same effect in all classrooms and with all students.
If we want to do a better job of evaluating the impact of EdTech on student outcomes, we need to understand how the technology is being used. We can’t assume that every teacher will use it in the same way or with the same effectiveness. We also need to challenge the assumption that technologies work consistently across all contexts. This assumption, which Professor Neil Selwyn from Monash University has described as technological essentialism, underestimates the importance of human agency and contextual variation.
My doctorate proposed a new approach for describing and measuring teacher proficiency with technology, one that is directly tied to student learning outcomes. If we can better understand the specific knowledge and skills teachers need to use technology effectively, and if we can measure these, we can begin to make clearer connections between technology use and learner outcomes. This kind of understanding can help explain why certain technologies have a positive impact, and under what conditions they lead to improved learning.
Unpacking the influence of teachers on EdTech learning
Consider a hypothetical example of a small-scale evaluation of a new technology: a software platform, a hardware tool, or perhaps the integration of generative AI into teaching. A pilot study with a small group of well-resourced teachers might show promising results. But when the same technology is rolled out more broadly, the anticipated benefits fail to materialise.
This is where measuring teacher proficiency can make a difference. By mapping the skills of participating teachers to the outcomes achieved by their students, we can begin to see when and why the technology works or doesn’t. The evidence may reveal, for example, that the technology only supports learning when teachers demonstrate certain competencies, or when they use it in particular ways.
This approach could help schools and systems assess both the potential of a technology and the professional learning required for it to be effective. Rather than asking whether a technology ‘works’ in general, we can begin to describe the conditions under which technologies can contribute to better learning.
Measuring the impact of EdTech
Measuring the impact of EdTech has proved challenging over the past 2 decades. Many schools and teachers are left asking for clearer, more practical guidance. Large-scale studies and meta-analyses tell us that EdTech has some overall benefit, but do not account for differences in teacher practice or local context. On the other hand, individual studies may suggest positive effects in specific settings but don’t always scale. Certification programs and rating systems help, but they depend on the participation of companies, the judgment of reviewers, and EdTech remaining static, which it won’t.
A more promising path forward lies in focusing on teacher knowledge and practice: understanding the capabilities educators need to use technology in ways that support learning. If we can describe and assess those capabilities, we can support better teaching, drive more effective use of technology, and ultimately deliver better outcomes for students.
References
Falck, O., Mang, C., & Woessmann, L. (2018). Virtually no effect? Different uses of classroom computers and their effect on student achievement. Oxford Bulletin of Economics and Statistics, 80(1), 1–38.
Hattie, J. (no date). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Retrieved from www.visiblelearningmetax.com.
Mergoni, A., Sonchin, M., & Agasisti, T. (2023). The effect ICT on schools’ efficiency: Empirical evidence on 23 European countries. Omega, 119 (102891).
What kinds of evidence or information have you used when deciding to introduce a new technology? Did you draw on research findings, colleague recommendations, student feedback, product reviews, or your own trial and error?
Have you ever used a technology because it was recommended or widely adopted, only to find it didn’t work well in your context? What factors might have contributed to that?
