Data-Based Decision-Making for School Improvement: Research Insights and Gaps

Introduction and Overview of Data-Based Decision-Making

Data-based decision-making (DBDM), or data use, is a critical process for addressing complex problems in schools that lack obvious solutions. Acting quickly on issues without data might feel efficient, but it is often ineffective. For example, a school might invest in expensive curriculum materials to improve student achievement, but if the actual cause is a lack of targeted support for specific students, the investment is wasted and the problem may be exacerbated. Using data allows schools to determine the actual causes of problems before implementing improvement actions.

Research indicates that data use can contribute to increased student learning and achievement (Lai et al. $2014$ ; McNaughton, Lai, and Hsaio $2012$ ; Poortman and Schildkamp $2016$ ; Van Geel et al. $2016$ ). This theoretical paper by Kim Schildkamp ( $2019$ ) explores recent literature, including review studies by Datnow and Hubbard ( $2015$ , $2016$ ), Heitink et al. ( $2016$ ), Hoogland et al. ( $2016$ ), and Schenke and Meijer ( $2018$ ), to provide a definitive model of school improvement through data.

The Iterative Process of School Improvement

School improvement is conceptualized as an iterative process where data use plays a central role. The author proposes a model consisting of four main components orbiting a central goal-setting phase:

Goal Setting: The foundational step that informs all other processes.
Collecting Data: Gathering relevant evidence related to the goals.
Sense-Making: Analyzing and interpreting data to find meaning and causes.
Action and Evaluation: Implementing concrete changes and assessing if goals were achieved.

This cycle is continuous; evaluative data leads back to the re-evaluation of goals or the setting of new ones.

Component $1$ : Goal Setting

Goal setting is placed at the apex of the iterative model because data use must be purposeful rather than starting with the data itself. Goals should be concrete, measurable, and connected to the quality of teaching and learning.

Levels of Goals

Goals operate across multiple levels of the educational system:

Student and Classroom Level: Specific student learning goals.
School Level: Aggregated achievement goals for the whole institution.
System Level: Benchmarks and educational standards set by local, regional, or national policy.

Stakeholders and Values

Goals are never value-neutral; they reflect assumptions about what is important to learn. They are often the result of negotiation and debate between stakeholders. School leaders play a vital role by translating broad policies into specific school goals, prioritizing certain areas, and ensuring a collective dialogue exists within the school culture.

Three Blocks of Research-Identified Goals

Previous research (e.g., Schildkamp, Karbautzki, and Vanhoof $2014$ ) divides school goals into three categories:

Accountability Goals: Using data (e.g., assessment results) to demonstrate progress to evaluation bodies and parents.
School Development Goals: Monitoring and improving school functions, such as curriculum development and professional development (PD) planning.
Instructional Goals: Improving classroom quality, such as setting individual learning goals and providing student feedback.

Tensions in Goal Setting

There is an inherent tension between improvement goals and accountability goals. Over-reliance on accountability can lead to "gaming the system," "teaching to the test," or excluding weaker students from testing to maintain status (Ehren and Swanborn $2012$ ). However, when used meaningfully, accountability makes a system transparent and reveals areas needing improvement.

Component $2$ : Collecting Data

To avoid over-reliance on narrow assessment data, schools must triangulate multiple sources. The paper proposes a broad, inclusive definition of "evidence-informed practice" that includes four categories of data:

$1.$ Formal Data

Defined by Schildkamp and Kuiper ( $2010$ , $482$ ) as the process of "systematically analyzing existing data sources within the school, applying the outcomes of analyses in order to innovate teaching, curricula, and school performance." It includes:

Assessment results.
Surveys.
Structured classroom observations.

$2.$ Informal Data

This includes information collected by teachers in everyday practice, often described as "professional judgment" or "intuitive data." Examples include:

Everyday observations of students.
Dialogue and conversations within an "assessment-for-learning" approach.
Peer-to-peer discussions.

$3.$ Research Results

Defined by Flood and Brown ( $2018$ ) as the process of teachers "accessing, evaluating and applying the findings of academic research." This includes:

Practitioner/Action Research: Research conducted by teachers in their own schools.
Scientific Research: Findings from external studies (whether the school participated or not).

$4.$ Big Data

Characterized by the "three Vs" (Laney $2001$ ):

Volume: Huge amounts of data.
Variety: Data in varied forms.
Velocity: Data being continuously updated. Big data can be used to monitor and predict organizational performance (Veldkamp et al. $2017$ ).

Challenges in Data Collection

Data collection is socially constructed and never value-free. Organizations may suffer from "goal displacement," where they focus only on concepts that are easy to measure (e.g., test scores) while ignoring complex $21$ st-century goals such as motivation, critical thinking, or well-being.

Component $3$ : Sense-Making

Sense-making is the process of analyzing and interpreting data to identify problems and their causes. It is not a purely rational process; it is filtered through the lenses of personal experience, intuition, and pre-existing beliefs (Weick $1995$ ; Vanlommel et al. $2017$ ).

Barriers to Effective Sense-Making

Confirmation Bias: Fitting data into a frame that confirms prior assumptions.
Limited Data Triangulation: Drawing conclusions from an insufficient data set.
Heuristics: Using quick, simple strategies that require less cognitive effort but lead to false interpretations.
Difficulty in Translation: Teachers often struggle to turn data analysis into a concrete action plan.

Stakeholder Perspectives in Sense-Making

Policy-Makers: Often focus on standardized assessment data and familiar sources.
Principals: Need to read contextual circumstances to act responsively.
Teachers: Tend to rely more on informal data than formal data.
Students: Often overlooked, but they can use data to steer their own learning.

Component $4$ : Action and Evaluation

Sense-making should lead to improvement actions across three pillars: Curriculum (coherence), Assessment (formative), and Instruction (targeted support).

Five Types of Data Use

Research identifies different ways data is utilized in schools (Farley-Ripple et al. $2018$ ; Weiss $1998$ ):

Instrumental Use: Actually making changes in school and classroom practices.
Conceptual Use: Changing the thinking of teachers and leaders without immediate concrete action.
Strategic Use: Manipulating data to attain power or personal goals.
Symbolic Use: Complying with external pressure without meaningful engagement.
Misuse/Abuse: "Teaching to the test" or focusing only on students near a specific benchmark.

The Evaluation Phase

Evaluation brings the cycle full circle by asking:

Were the actions implemented?
Did they lead to the desired effects?
Was the original goal reached? This requires the collection of new data, creating a continuous loop of improvement.

Enablers, Barriers, and Research Gaps

Key Enablers

Data Literacy: The skills required to analyze and interpret various data types.
Leadership: Principals must monitor, model, scaffold, and encourage data use.
Distributive Leadership: Empowering teachers to take action based on data.
Resource Allocation: Providing time and access to high-quality tools (dashboards, data warehouses).

Identified Research Gaps

Multidisciplinary Integration: Combining expertise from technology, data mining, and psychology.
Cross-Sector Comparisons: Comparing educational data use to evidence-based medicine (Sackett et al. $1996$ ).
High-Tech vs. Human Touch: Investigating how digital tools and AI can support student responsibility for learning.
Big Data Ethics: Addressing who has access to data and the social/ethical implications of predictive modeling.
Sustainability: Moving beyond initial implementation to making data use an "organizational routine."

The Data Team Intervention

Schildkamp and Poortman ( $2015$ ) developed an effective PD intervention: the "Data Team."

Structure: Teams of educators use data collaboratively to solve a specific school problem.
Impact: Increased data literacy and improved student achievement across The Netherlands, Sweden, Belgium, England, and the USA.
Key Features of Effective PD: Creating protocols, providing long-term support, and making explicit links between data and instruction.

The article defines data broadly, categorizing it into four types for use in data-based decision-making (DBDM):

Formal Data: This encompasses systematically analyzed existing data sources within the school, which includes assessment results, surveys, and structured classroom observations.
Informal Data: Refers to the information collected by teachers through everyday practice, often described as professional judgment or intuitive data. It includes everyday observations of students and dialogues within an assessment-for-learning approach.
Research Results: These involve findings from academic research that teachers can access, evaluate, and apply. This covers both practitioner/action research conducted by teachers in their own contexts and scientific research findings from external studies.
Big Data: Characterized by the three Vs—volume (large amounts), variety (data in varied forms), and velocity (continuously updated). Big data is used to monitor and predict organizational performance.

Teachers utilize various types of data in their practice to enhance educational decision-making and improve student outcomes. Here’s how each type of data is used:

1. Formal Data

Assessment results: Teachers analyze standardized test scores and formative assessments to gauge student learning and identify areas needing attention.
Surveys: Feedback collected from students and parents can inform teaching strategies and areas for improvement.
Structured classroom observations: These provide evidence of teaching effectiveness and student engagement.

2. Informal Data

Everyday observations of students: Teachers rely on their intuitive judgment of students’ behaviors and learning progress during daily activities.
Dialogue and conversations: Informal interactions with students can highlight their understanding and misconceptions.
Peer-to-peer discussions: Collaborating with colleagues allows teachers to share insights gleaned from informal observations and improve practices collaboratively.

3. Research Results

Practitioner/Action research: Teachers conduct their own studies to assess the impact of specific strategies in their classrooms.
Scientific research findings: Educators apply insights from external studies to refine their teaching methodologies and curricular approaches.

4. Big Data

Monitoring and predicting: Teachers may use big data analytics to understand trends in student performance at a broader level, helping to tailor instruction and interventions.

Types of Data Teachers Rely On the Most

Teachers tend to rely heavily on informal data such as everyday observations and dialogue, as these provide immediate feedback and insights into student learning. However, they also value formal data from assessments to track progress and make informed decisions about instruction and interventions. Research results can supplement these practices but are often used less frequently due to time constraints and the accessibility of relevant studies.

Teachers utilise various data types—formal (e.g., assessments and surveys), informal (e.g., observations and peer discussions), research results, and big data—to enhance educational practices and improve student outcomes. Formal data evaluates student learning, while informal data provides immediate insights. Although educators rely primarily on informal data for its immediacy, formal data is also important for a comprehensive understanding of student progress. This interplay is crucial for effective data-based decision-making in schools (Schildkamp, 2019).