AI Feedback as a Complement to Professional Learning

Formative feedback grounded in teachers’ practices can enhance instruction and improve student outcomes (Kraft, Blazar, & Hogan, 2018; Shute, 2008; Steinberg & Sartain, 2015; Taylor & Tyler, 2012). Instructional coaches or mentor teachers often provide such feedback by observing classrooms, guiding teacher reflections and offering improvement suggestions (Kraft et al., 2018).

 

However, expert coaching is expensive and time-consuming, limiting most teachers’ access to consistent, high-quality feedback. In the United States, only ∼40% of schools provide teachers access to an instructional coach (Taie & Goldring, 2017) and in many schools, teachers primarily receive feedback from their principals, who often lack the time and knowledge to support teachers’ thorough analysis and synthesis of evaluation data (Firestone & Donaldson, 2019; Rigby et al., 2017).

 

Technology has emerged as a promising way to fill the gaps in teacher professional learning by providing teachers with data-driven opportunities to facilitate instructional improvement. Computerized tools can help teachers refresh their pedagogical content knowledge and rehearse in simulated environments (Copur-Gencturk, Li, Cohen, & Orrill, 2024; Markel, Opferman, Landay, & Piech, 2023), better respond to students’ written explanations in between class sessions (Bywater et al., 2019, Bywater et al., 2023), and reflect on recordings of their instruction through video (Chen, 2020) or text analytics (Demszky, Liu, Hill, Jurafsky, & Piech, 2023; Jacobs et al., 2022). Different from tools that provide computer-scaffolded instruction directly to students (e.g., Cognitive Tutor; Anderson, Corbett, Koedinger, and Pelletier (1995)), these teacher-facing tools represent the potential of computer technology to influence human-to-human instructional quality broadly through numerous mechanisms.

 

Using natural language processing (NLP), tools that provide formative feedback to teachers based on their instruction may complement human observation and coaching, improving instructional practice (Jacobs, Scornavacco, Clevenger, Suresh, & Sumner, 2024) and even student outcomes in online settings (Demszky & Liu, 2023; Demszky et al., 2023).

 

Such automated feedback tools take a recording of a teacher’s lesson as input, transcribe and analyze the recording to identify high-leverage instructional practices, and deliver insights to the teacher to facilitate reflection and instructional improvement. Since such feedback is cost-efficient, scalable, and can be delivered privately, quickly, and frequently, researchers and technology providers (e.g. TeachFX, EdThena) are seeking to understand how they can be best put to teachers’ service.

 

Despite the need for scalable K-12 classroom observation and feedback tools as well as encouraging studies with such tools in online teaching settings, to our knowledge, there exists no rigorous experimental evaluation of whether automated feedback might work in K-12 in-person learning contexts. To this end, we present results from an experiment in which we provided teachers with feedback related to focusing questions — a high-leverage teaching practice involving asking questions that probe student thinking and encourage students to reflect on their thoughts and those of their classmates (Alic et al., 2022; Herbel-Eisenmann & Breyfogle, 2005; National Council of Teachers of Mathematics, 2014; Wood, 1998). Our experimental design allows us to causally estimate whether providing teachers feedback about focusing questions increases the number of such questions and whether it yields related improvements in instruction, such as increasing the amount of student talk and student reasoning.

 

We also add to the literature on human-computer interaction by seeking to understand how teachers engage with and perceive the utility of our automated feedback. Prior research on technology integration in classrooms indicates that teachers’ perceived utility of the technology has a strong influence on their technology adoption (Backfisch et al., 2021, Ertmer et al., 2012, Fütterer et al., 2023, Kale, 2018, Lachner et al., 2021, Scherer et al., 2019, Wang and Zhao, 2023). This suggests that teachers need to see the value of receiving automated feedback on their instruction in order to effectively use the tool. Jacobs et al., 2022, Jacobs et al., 2024 have found that many teachers see automated feedback as a valuable vehicle for self-reflection, but that perceptions of accuracy can impact their engagement with such feedback. We seek to deepen this knowledge about factors that may impact teachers’ perception of and engagement with automated feedback by conducting qualitative interviews with a subset of teachers in the experimental study.

 

Thus this mixed-method study is the first that combines a pre-registered randomized controlled trial and qualitative interviews to experimentally test the impact of and describe K-12 teacher engagement with automated feedback on instruction. In doing so, we address the following research questions:

 

 

To answer these questions, we partnered with TeachFX, a company that delivers feedback to teachers based on classroom recordings via a phone application. We leveraged TeachFX’s newly established partnership with the state of Utah to facilitate professional learning for mathematics and science teachers. We randomly assigned teachers to a treatment or control condition based on whether they received automated feedback on focusing questions. We collected recordings of their instruction, post-treatment surveys, and interview data to understand the impact of the treatment as well as teachers’ engagement with and perceptions of automated feedback.

 

In the following sections, we begin with an overview of related work on technology-based professional learning for teachers and teachers’ technology integration. Subsequently, we provide a background for our current study (Sections 3 and 4), with details on the technology platform and participants. In Section 5, we describe the experimental design, including the randomized setup, study procedures and the interview protocol. In Section 6, we provide an overview of the approach we took to answer each research question.

 

In Section 7, we provide the results of our research questions. We conclude by discussing the implications of these results for both research and practice related to using computerized tools in teacher professional learning.