Evaluation of the Humanities Curriculum Project: A Wholistic Approach

Overview

This document is an interim report on an evaluation exercise.
It outlines the characteristics of the curriculum intervention, its impact on the educational system, and the design of an appropriate evaluation.
The evaluation attempts to convey the empirical roots and the scope of its concerns.

Project Background

The Humanities Curriculum Research and Development Project was established in 1967 to prepare for the raising of the school-leaving age from 15 to 16 in 1972.
The project is jointly funded by the Nuffield Foundation and the Schools Council.
The evaluation unit is financed solely by the Schools Council, with a four-year life from 1968 to 1972.
The total budget for the Project and evaluation is approximately a quarter of a million pounds.
The central team's role was to provide stimulus, support, and materials for teachers and schools teaching humanities to students aged 14 to 16.
Humanities was defined as the study of important human issues, focusing on work in controversial areas.

Project Aims and Approach

The project aimed to address the demand for relevant curriculum for adolescents and to face controversial issues honestly.
The problem was to allow adolescents to reach views responsibly without teacher bias or undue peer pressure.
They tried to stimulate small group discussions using evidence like prose, verse, drama, photographs, paintings, and tapes.
The teacher's role was as a neutral chairman and resource consultant.
Collections of materials were produced in areas such as war, education, relations between the sexes, family, poverty, people and work, race, living in cities, and law and order.
From 1968 to 1970, the first collections were used experimentally in thirty-six schools throughout England and Wales.
Teachers tested suggested rules for discussion to shape research and creative activities around the discussion.
From Easter 1970, revised packs were commercially published, and training schemes for teachers were set up.
During the current academic year, some four to five hundred schools are using the materials.

Evaluation Design and Implementation

In August 1968, an evaluator was hired to study the project during its two-year trial period and design an evaluation for implementation in 1970 to 1972.
The Project Director asked for an independent evaluation.

Key Characteristics of the Project

The project stressed teacher responsibility and judgment, rather than dispensing expertise from a position of authority.
The central team aimed for a relationship of equality with participating personnel, inviting teachers to become technologists of curriculum rather than technicians.
The project adopted 'understanding' as the aim, reflecting a faith in educational rather than social adjustment approaches to controversial issues.
The design started with content specification, followed by aim formulation, analysis into process criteria, and an experiment to realize a pedagogically effective 'form' embodying these criteria.
The project's success was highly dependent on the teacher's understanding of philosophic and pedagogic principles for intelligent classroom practice.

Initial Evaluation Concerns

The evaluator was appointed to evaluate an innovation that seemed likely to fail, requiring induction courses, being difficult to use, costly, and conflicting with established values.
The project showed promise as a case study in the pathology of innovation.
Basing criteria exclusively on short-term pupil learnings was deemed inappropriate for a radical intervention in the organizational structure of a school system.

Paramount Considerations for Evaluation Approach

The sponsor of the evaluation was a government agency with responsibility for national curriculum development.
There was a need for information to aid planning at the national level, with a focus on patterns of interaction within the system initiated by project inputs.
Because the central team shared decision-making with participating schools, divergence in institutional response was anticipated.
Field studies of the project in various operational contexts were needed.
The prevailing climate of educational theory favored the behavioral objectives model.
The aim was to describe the project in a form accessible to public and professional judgment.
Evaluation design, strategies, and tactics would evolve in response to the project's impact on the system.

Project in Experimental Schools: 1968/70

The 36 schools were nominated by their administering authorities, reflecting differences in judgment and priorities.
Participating teachers met with the central team at regional conferences to explain the experiment.
The experimental schools embraced a wide range of environmental, compositional, structural, and cultural variables.
Contextual diversity was compounded by differences in decisions about introduction, organization, and implementation.
Differences among participating personnel included motivation, commitment, understanding, and expectations.
The extent and nature of support from the local authority also varied.
The immediate impact of the project was alarming, with enormous confusion and misunderstanding.
There were many unanticipated problems and widespread misperception of the demands that the project was making.

Specific Problems and Misconceptions

The importance of headmasters in innovation was underestimated.
The project was seen to be manipulating major variables in the school, including established patterns of social control.
The inflexible administrative institutions could not easily create the necessary conditions for the experiment.
Teachers did not anticipate the extent to which many students had developed a trained incapacity for this work.
Teachers didn't anticipate the depth of alienation from any kind of curriculum offering, nor the degree to which they and their students had been socialized into a tradition of teacher dominance and custodial attitudes.
Many became locked in role conflicts, others in attempts to bridge an unforeseen credibility gap between themselves and their students.
The central team failed to communicate successfully the nature of the enterprise.
From the teacher's point of view, the ethos of the Project was evangelical rather than exploratory, and the suggested teaching strategies tests of teacher proficiency rather than research hypotheses.
Many felt on trial, reducing their capacity to profit from the experience and adversely affecting their feedback to the Centre.

Diverse Outcomes

Although the program proved generally demanding, difficult, and disturbing, there were striking exceptions and many contradictions.
Limited explanations of perceived failure/success, such as pupil ability, teacher behavior, or institutional ethos, could not be readily generalized.
The evaluator concentrated on establishing what was happening in the schools and gathering information to explain differing patterns of action and response.
The evaluator studied the behavior of the central team and the interaction between them, the local authorities, and the schools.
Data was gathered for each school about the external forces of support and opposition that were mobilized by the Project's intervention.
A checklist of hard and soft data items was gathered to add up to an institutional profile of each school.
Teachers' understanding of Project theory and attitudes towards it was assessed by questionnaires.
Teachers sent in audiotapes of their classroom discussions together with written supplementary data.

School Visits and Case Studies

The evaluator embarked on a series of visits to the schools, initially intending to study all of them at first hand.
After visiting about half of them, the plan was abandoned in favor of case-studying a small number.

Propositions Arising from the Studies

Human action in educational institutions differs widely because of the number of variable influences that determine it.
The impact of an innovation is not a set of discrete effects, but an organically related pattern of acts and consequences.
Curriculum interventions have many more unanticipated consequences than is normally assumed in development and evaluation designs.
No two schools are sufficiently alike in their circumstances that prescriptions of curricular action can adequately supplant the judgment of the people in them.
The goals and purposes of the program developers are not necessarily shared by its users.

Further Considerations in the Development of the Evaluation Design

Faced with a central team who were opposed to the use of 'objectives', an alternative concept of Evaluation was sought.
The possibility of defining responsibilities in relation to likely readers of the report was explored.
The idea of evaluation for consumers attracted the evaluator.
'Consumers' became redefined as decision-makers, including the sponsors, the local education authority, the schools, and the examination board.
The task of evaluation was then defined as that of answering the questions that decision-makers ask.
The task is now viewed as feeding the judgment of decision-makers by promoting understanding of the considerations that bear upon curricular action.
The current orientation is towards educing an empirical rather than a normative model of educational decision- making and its consequences.

Data Requirements and Focus

Decision-making groups differ in their data requirements.
Individuals differ in the degree of confidence they place in different kinds of data.
The evaluation is integrating both subjective and objective approaches in a broad study of the Project from 1970 to 1972.
Much evaluation work in the past has been simplistic or too narrowly focused.
Education is a complex practical activity, and reducing that complexity to singularistic perspectives tends to distort the reality.
Wholistic evaluation designs can give a more adequate view of what we are trying to change, and of what is involved in changing it.

Systematic Attempt

A systematic attempt will be made to document the outcomes of the Project at the levels of system, institution, teacher, and pupil.
These outcomes will be located within a record of antecedent events and conditions.
Variations in outcome will be explained in terms that will lead to more informed curriculum decisions.

Evaluation Design Components

In a large sample of schools (c. 100):
- Gathering input, contextual, and implementation data by questionnaire.
- Gathering experiential, impressionistic, and judgmental data from teachers and pupils.
- Objective measurement of pupil and teacher change (pre-tests on 21 objective instruments).
- Tracing process variation by multiple-choice feedback instruments.
- Monitoring institutional response by semi-structured diary instruments.
In a small sample of schools (c. 12):
- Case studies of patterns of decision-making, communication, training, and support in local areas.
- Case studies of individual schools within these areas.
- Study of process dynamics by audiotape, videotape, and observation.

Summary

The job is to describe the experience of hundreds of schools embarking on work with Project materials.
The description must be helpful to those who have to make judgments in this field.
In an open-ended inquiry program, where the aims are broadly defined and teacher behavior is a matter of individual response, this is no simple matter.
There is a need for both qualitative field studies and quantitative data gathering and measurement techniques to be combined.
By interweaving these studies, the aim is to advance understanding of the interplay of forces in this curriculum innovation.