Data-Based Decision Making by Teachers in K-12 Schools: A Scoping Review

Areej Tayem; Isabelle Bourgeois

doi:10.21432/cjlt28781

Authors

Areej Tayem University of Ottawa
Isabelle Bourgeois University of Ottawa

DOI:

https://doi.org/10.21432/cjlt28781

Keywords:

K-12 Education, Teacher Practices, student outcomes, data-based decision making

Abstract

Despite the widespread adoption of data-based decision making (DBDM) policies in schools around the world, there is limited understanding of how teachers use DBDM in K-12 classrooms and the impact of DBDM training on teacher practices and student outcomes. This scoping review aims to provide an overview of the existing literature on the uses of DBDM by teachers globally and identify gaps in the field. The findings (a) highlight a geographical and temporal clustering, with a notable emphasis on studies conducted in the United States and the Netherlands and published in 2016–2017 and 2020–2022; (b) identify a gap in the literature, particularly in the context of online and secondary schools, where the predominant focus has been on elementary and in-person settings; and (c) suggest that although DBDM interventions have been found helpful in altering teacher practices and student outcomes, there is still a need for more sustainable support to enhance DBDM implementation. The study concludes with recommendations for future DBDM research, building on implications from previous interventions.

Author Biographies

Areej Tayem, University of Ottawa

Areej Tayem is a Ph.D. candidate and part-time Professor in the Faculty of Education at the University of Ottawa in Canada. Her research focuses on learning analytics and data-based decision making (DBDM) in K-12 education. Email: atayem@uottawa.ca

Isabelle Bourgeois, University of Ottawa

Isabelle Bourgeois is a full Professor in the Faculty of Education at the University of Ottawa in Canada. Her main research activities focus on program evaluation and evaluation capacity building in public and community organizations. Email: isabelle.bourgeois@uottawa.ca

References

Abdusyakur, I., & Poortman, C. L. (2019). Study on data use in Indonesian primary schools. Journal of Professional Capital and Community, 4(3), 198–215. https://doi.org/10.1108/JPCC-11-2018-0029

Admiraal, W., Vermeulen, J., & Bulterman-Bos, J. (2020). Teaching with learning analytics: How to connect computer-based assessment data with classroom instruction? Technology, Pedagogy and Education, 29(5), 577–591. https://doi.org/10.1080/1475939X.2020.1825992

Andersen, I. G. (2020). What went wrong? Examining teachers’ data use and instructional decision making through a bottom-up data intervention in Denmark. International Journal of Educational Research, 102, 101585. https://doi.org/10.1016/j.ijer.2020.101585

Behrens, J., Piety, P., DiCerbo, K., & Mislevy, R. (2018). Inferential foundations for learning analytics in the digital ocean. In D. Niemi, R. D. Pea, B. Saxberg, & R. E. Clark (Eds.), Learning analytics in education (pp. 1–48). Information Age Publishing Inc.

Campos, F. C., Ahn, J., DiGiacomo, D. K., Nguyen, H., & Hays, M. (2021). Making sense of sensemaking: Understanding how K-12 teachers and coaches react to visual analytics. Journal of Learning Analytics, 8(3), 60–80.

Carlson, D., Borman, G. D., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33(3), 378–398. https://doi.org/10.3102/0162373711412765

Cheng, L. (1999). Changing assessment: Washback on teacher perspectives and actions. Teaching and Teacher Education, 15(3), 253–271. https://doi.org/10.1016/S0742-051X(98)00046-8

Cheng, L., & Curtis, A. (2004). Washback or backwash: A review of the impact of testing on teaching and learning. In L. Cheng, Y. Watanabe, & A. Curtis (Eds.), Washback in language testing: Research contexts and methods (pp. 3–17). Lawrence Erlbaum.

Christman, J. B., Ebby, C. B., & Edmunds, K. A. (2016). Data use practices for improved mathematics teaching and learning: The importance of productive dissonance and recurring feedback cycles. Teachers College Record, 118(11), 1–32. https://doi.org/10.1177/016146811611801101

Copp, D. T. (2017). Policy incentives in Canadian large-scale assessment: How policy levers influence teacher decisions about instructional change. Education Policy Analysis Archives, 25, 115. https://doi.org/10.14507/epaa.25.3299

Curry, K. A., Mwavita, M., Holter, A., & Harris, E. (2016). Getting assessment right at the classroom level: Using formative assessment for decision making. Educational Assessment, Evaluation and Accountability, 28, 89–104. https://doi.org/10.1007/s11092-015-9226-5

Datnow, A., & Hubbard, L. (2015). Teachers’ use of assessment data to inform instruction: Lessons from the past and prospects for the future. Teachers College Record, 117(4), 1–26. https://doi.org/10.1177/016146811511700408

Datnow, A., Choi, B., Park, V., & John, E. S. (2018). Teacher talk about student ability and achievement in the era of data-driven decision making. Teachers College Record, 120(4), 1–34. https://doi.org/10.1177/016146811812000408

Datnow, A., Lockton, M., & Weddle, H. (2021). Capacity building to bridge data use and instructional improvement through evidence on student thinking. Studies in Educational Evaluation, 69, 100869. https://doi.org/10.1016/j.stueduc.2020.100869

De Simone, J. J. (2020). The roles of collaborative professional development, self-efficacy, and positive affect in encouraging educator data use to aid student learning. Teacher Development, 24(4), 443–465. https://doi.org/10.1080/13664530.2020.1780302

Dunn, K. E., Airola, D. T., Lo, W. J., & Garrison, M. (2013). What teachers think about what they can do with data: Development and validation of the data driven decision-making efficacy and anxiety inventory. Contemporary Educational Psychology, 38(1), 87–98. https://doi.org/10.1016/j.cedpsych.2012.11.002

Ebbeler, J., Poortman, C. L., Schildkamp, K., & Pieters, J. M. (2016). Effects of a data use intervention on educators’ use of knowledge and skills. Studies in Educational Evaluation, 48, 19–31. https://doi.org/10.1016/j.stueduc.2015.11.002

Faber, J., Glas, C., & Visscher, A. J. (2018). Differentiated instruction in a data-based decision-making context. School Effectiveness and School Improvement, 29(1), 43–63. https://doi.org/10.1080/09243453.2017.1366342

Farley-Ripple, E. N., Jennings, A. S., & Buttram, J. (2019). Toward a framework for classifying teachers’ use of assessment data. AERA Open, 5(4), 2332858419883571. https://doi.org/10.1177/2332858419883571

Farrell, C. C., & Marsh, J. A. (2016a). Metrics matter: How properties and perceptions of data shape teachers’ instructional responses. Educational Administration Quarterly, 52(3), 423–462. https://doi.org/10.1177/0013161X16638429

Farrell, C. C., & Marsh, J. A. (2016b). Contributing conditions: A qualitative comparative analysis of teachers’ instructional responses to data. Teaching and Teacher Education, 60, 398–412. https://doi.org/10.1016/j.tate.2016.07.010

Filderman, M. J., Austin, C. R., & Toste, J. R. (2019). Data-based decision making for struggling readers in the secondary grades. Intervention in School and Clinic, 55(1), 3–12. https://doi.org/10.1177/1053451219832991

Gelderblom, G., Schildkamp, K., Pieters, J., & Ehren, M. (2016). Data-based decision making for instructional improvement in primary education. International Journal of Educational Research, 80, 1–14. https://doi.org/10.1016/j.ijer.2016.07.004

Hebbecker, K., Förster, N., Forthmann, B., & Souvignier, E. (2022). Data-based decision-making in schools: Examining the process and effects of teacher support. Journal of Educational Psychology, 114(7), 1695. https://psycnet.apa.org/doi/10.1037/edu0000530

Heinrich, C., & Good, A. (2018). Research-informed practice improvements: Exploring linkages between school district use of research evidence and educational outcomes over time. School Effectiveness and School Improvement, 29(3), 418–445. https://doi.org/10.1080/09243453.2018.1445116

Ho, J. E. (2022). What counts? The critical role of qualitative data in teachers’ decision making. Evaluation and Program Planning, 91, 102046. https://doi.org/10.1016/j.evalprogplan.2021.102046

Hoover, N. R., & Abrams, L. M. (2013). Teachers’ instructional use of summative student assessment data. Applied Measurement in Education, 26(3), 219–231. https://doi.org/10.1080/08957347.2013.793187

Kempf, A. (2015). The school as factory farm: All testing all the time. In The pedagogy of standardized testing (pp. 13–28). Palgrave Macmillan US. https://doi.org/10.1057/9781137486653_2

Keuning, T., Geel, M. V., Visscher, A., Fox, J.-P., & Moolenaar, N. M. (2016). The transformation of schools’ social networks during a data-based decision making reform. Teachers College Record, 118(9), 1–33. https://doi.org/10.1177/016146811611800908

Lai, M. K., & McNaughton, S. (2016). The impact of data use professional development on student achievement. Teaching and Teacher Education, 60, 434–443. https://doi.org/10.1016/j.tate.2016.07.005

Lai, M. K., Wilson, A., McNaughton, S., & Hsiao, S. (2014). Improving achievement in secondary schools: Impact of a literacy project on reading comprehension and secondary school qualifications. Reading Research Quarterly, 49(3), 305–334. https://doi.org/10.1002/rrq.73

Maier, U. (2010). Accountability policies and teachers’ acceptance and usage of school performance feedback – A comparative study. School Effectiveness and School Improvement, 21(2), 145–165. https://doi.org/10.1080/09243450903354913

Marsh, J. A. (2012). Interventions promoting educators’ use of data: Research insights and gaps. Teachers College Record, 114(11), 1–48. https://doi.org/10.1177/016146811211401106

Marsh, J. A., Bertrand, M., & Huguet, A. (2015). Using data to alter instructional practice: The mediating role of coaches and professional learning communities. Teachers College Record, 117(4), 1–40. https://doi.org/10.1177/016146811511700411

Marsh, J. A., Pane, J. F., & Hamilton, L. S. (2006, Nov 7). Making sense of data-driven decision making in education [Occasional Paper]. RAND Corporation.

Michaud, R. (2016). The Nature of Teacher Learning in Collaborative Data Teams. Qualitative Report, 21(3), 529-545. https://doi.org/10.46743/2160-3715/2016.231.

Park, V., & Datnow, A. (2017). Ability grouping and differentiated instruction in an era of data-driven decision making. American Journal of Education, 123(2), 281–306. https://doi.org/10.1086/689930

Peters, M. T., Förster, N., Hebbecker, K., Forthmann, B., & Souvignier, E. (2021). Effects of data-based decision-making on low-performing readers in general education classrooms: Cumulative evidence from six intervention studies. Journal of Learning Disabilities, 54(5), 334–348. https://doi.org/10.1177/00222194211011580

Rangel, V. S., Bell, E. R., & Monroy, C. (2017). A descriptive analysis of instructional coaches’ data use in science. School Effectiveness and School Improvement, 28(2), 217–241. https://doi.org/10.1080/09243453.2016.1255232

Reed, D. K. (2015). Middle level teachers’ perceptions of interim reading assessments: An exploratory study of data-based decision making. RMLE Online: Research in Middle Level Education, 38(6), 1–13. https://doi.org/10.1080/19404476.2015.11462119

Regan, K., Evmenova, A. S., Mergen, R. L., Verbiest, C., Hutchison, A., Murnan, R., Field, S., & Gafurov, B. (2023). The feasibility of using virtual professional development to support teachers in making data‐based decisions to improve students’ writing. Learning Disabilities Research & Practice, 38(1), 40–56. https://doi.org/10.1111/ldrp.12301

Rodríguez‐Martínez, J. A., González‐Calero, J. A., del Olmo‐Muñoz, J., Arnau, D., & Tirado‐Olivares, S. (2023). Building personalised homework from a learning analytics based formative assessment: Effect on fifth‐grade students’ understanding of fractions. British Journal of Educational Technology, 54(1), 76–97. https://doi.org/10.1111/bjet.13292

Schifter, C., Natarajan, U., Ketelhut, D. J., & Kirchgessner, A. (2014). Data-driven decision-making: Facilitating teacher use of student data to inform classroom instruction. Contemporary Issues in Technology and Teacher Education, 14(4), 419–432.

Schildkamp, K., & Ehren, M. (2013). From “intuition” to “data”-based decision making in Dutch secondary schools? In K. Schildkamp, M. K. Lai, & L. Earl (Eds.), Data-based decision making in Education (pp. 49–67). Springer Netherlands. https://doi.org/10.1007/978-94-007-4816-3_4

Schildkamp, K., Poortman, C., & Handelzalts, A. (2016). Data teams for school improvement. School Effectiveness and School Improvement, 27(2), 228–254. https://doi.org/10.1080/09243453.2015.1056192

Schildkamp, K., Poortman, C., Luyten, H., & Ebbeler, J. (2017). Factors promoting and hindering data-based decision making in schools. School Effectiveness and School Improvement, 28(2), 242–258. https://doi.org/10.1080/09243453.2016.1256901

Shirley, D. (2017). The new imperatives of educational change: Achievement with integrity. Routledge. https://doi.org/10.4324/9781315682907

Staman, L., Timmermans, A. C., & Visscher, A. J. (2017). Effects of a data-based decision making intervention on student achievement. Studies in Educational Evaluation, 55, 58–67. https://doi.org/10.1016/j.stueduc.2017.07.002

Staman, L., Visscher, A. J., & Luyten, H. (2014). The effects of professional development on the attitudes, knowledge and skills for data-driven decision making. Studies in Educational Evaluation, 42, 79–90. https://doi.org/10.1016/j.stueduc.2013.11.002

Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467–485. https://doi.org/10.7326/M18-0850

Tsai, Y., Poquet, O., Gašević, D., Dawson, S., & Pardo, A. (2019). Complexity leadership in learning analytics: Drivers, challenges and opportunities. British Journal of Educational Technology, 50(6), 2839–2854. https://doi.org/10.1111/bjet.12846

Truckenmiller, A. J., Cho, E., & Troia, G. A. (2022). Expanding assessment to instructionally relevant writing components in middle school. Journal of School Psychology, 94, 28–48. https://doi.org/10.1016/j.jsp.2022.07.002

van den Bosch, R. M., Espin, C. A., Chung, S., & Saab, N. (2017). Data‐based decision‐making: Teachers’ comprehension of curriculum‐based measurement progress‐monitoring graphs. Learning Disabilities Research & Practice, 32(1), 46–60. https://doi.org/10.1111/ldrp.12122

van Geel, M., Keuning, T., Visscher, A. J., & Fox, J.-P. (2016). Assessing the Effects of a School-Wide Data-Based Decision-Making Intervention on Student Achievement Growth in Primary Schools. American Educational Research Journal, 53(2), 360–394. https://doi.org/10.3102/0002831216637346

van der Scheer, E. A., & Visscher, A. J. (2016). Effects of an intensive data-based decision making intervention on teacher efficacy. Teaching and Teacher Education, 60, 34–43. https://doi.org/10.1016/j.tate.2016.07.025

van der Scheer, E. A., Glas, C. A., & Visscher, A. J. (2017). Changes in teachers’ instructional skills during an intensive data-based decision making intervention. Teaching and Teacher Education, 65, 171–182. https://doi.org/10.1016/j.tate.2017.02.018

Ylimaki, B., & Brunderman, L., (2019). School development in culturally diverse U.S. schools: Balancing evidence-based policies and education values. Education Sciences, 9(84), 1–15. https://doi.org/10.3390/educsci9020084