Optimizing Schools Case Study Notes

Introduction to AI Ethics Case Study

The case study explores ethical dilemmas arising from the deployment of AI systems in society.
It is designed to stimulate discussion about issues at the intersection of AI and ethics.
The case study was developed from an interdisciplinary workshop series at Princeton University.
The collaboration involved the University Center for Human Values (UCHV) and the Center for Information Technology Policy (CITP) at Princeton.

The Situation at Minerva High School

Minerva High School in Pittsburgh, PA, faced a high student dropout rate, reaching 9% in 2012.
Only 55% of students graduated on time, compared to the state average of 76.4%.
The school board demanded that Principal Mr. Vulcani address the problem, threatening funding loss or shutdown.
Students were disengaged, and teachers were demotivated by shifting incentives and pedagogical approaches.

Data Collection and the Proposal for AI Intervention

The school collected extensive data on students, including performance, disciplinary, and attendance records.
Scannable ID cards tracked library access, class attendance, and purchases.
The Wi-Fi network monitored students' internet use and movements on campus.
The school board suggested using data science and machine learning to analyze this data and identify causes for the high dropout rates.
The goal was to identify predictors of disengagement, flag at-risk students, and suggest interventions.
Interventions included direct talks with students, workload adjustments, parental contact, and counseling.
Mr. Vulcani contracted Hephaestats, a local data science company, to implement AI solutions.
Hephaestats promised insights into business processes using AI.

Goals of Implementing AI

Identify predictors of student disengagement to flag at-risk students using machine learning.
Equip teachers with detailed information to allocate resources and suggest specific interventions.
Maintain transparency in the system's use.

Mr. Vulcani and the school board provided Hephaestats with existing and new student databases.
Students and parents were not notified or given the option to opt out, due to concerns about reaching consensus and the urgency of the situation.

Discussion Question #1

Who should make decisions about adopting AI technologies?
Who are the relevant stakeholders, and to what extent should they be involved?
How does not including an opt-out option affect the legitimacy of the school's actions?

Data Analysis by Hephaestats

Hephaestats conducted a broad data analysis, examining demographics, academic factors, and teacher statistics.
They correlated data from previous dropouts with information about current students to recognize patterns.
Hephaestats generated synthetic data through inferences derived from the school's original data.

Discussion Question #2

Did the school violate student privacy by sharing data with Hephaestats?
Is this breach justifiable, and on what grounds?
Would perceptions differ based on whether one is the parent of an at-risk student or a valedictorian?

Findings and Interventions

Hephaestats identified eight key indicators that predicted student dropout with 92% accuracy.
Reasons included administrative issues, external factors (job, domestic responsibilities), and previously unconsidered factors (poor nutritional options in the cafeteria).
Teachers received profiles of at-risk students with suggested interventions, such as tutoring or modifying assignments.
The administration adjusted the campus environment to encourage better behavior, such as changing course schedules and removing sugary foods from the cafeteria.
Teachers were instructed to provide more attention to struggling students.

Results and Praises

By the end of the 2016-17 academic year, Minerva High School showed significant improvement.
The four-year graduation rate rose from 55% to 85%.
The dropout rate decreased from 9% to 5%.
A higher percentage of students were accepted to four-year colleges.
Mr. Vulcani credited Hephaestats for the improvement in student outcomes.

Discussion Question #3

How do we define a successful outcome for Minerva High School?
What was the school hoping to optimize by contracting Hephaestats?
To what extent is this end legitimate?
Did Hephaestats ask the right questions and use the right proxies?

Discussion Question #4

Correlation doesn't necessarily imply causation.
What are some alternative explanations for the improvement in dropout rates?

Concerns and Criticisms

Students and parents raised concerns when they were informed about Hephaestats' involvement.
They were uncomfortable with the school's ongoing data collection and use policies.
Some teachers and administrators voiced criticisms of and opposition to the new system.

Ethical Objection #1: Privacy

Critics argued that the school provided vast amounts of data without regard for its sensitivity, amounting to a "fishing expedition."
They claimed the school breached privacy laws or norms by handing over a broad range of data to a commercial entity without consent.
Concerns were raised about setting a bad precedent for privacy violations.

Ethical Objection #2: Dehumanization of Students and Faculty

Students felt like research subjects being optimized in a living lab.
Critics argued the system prioritized improving the school's ranking over subtler aims, such as providing a safe space for failure and a nurturing environment.
Teachers felt their training, experience, and intuitions were being overridden by the AI system.

Ethical Objection #3: Transparency

Students, parents, and teachers felt forced to trust a process they could not scrutinize.
They wanted to be involved in the design process and requested a procedure for appealing disputed results and the source code of Hephaestats' systems.
They threatened to file a Freedom of Information Act (FOIA) request.

Skepticism About AI and the Consultant Effect

Teachers argued that the AI system was merely reiterating what they had been saying for years.
They attributed much of the improvement to the "Consultant Effect."
One math teacher questioned whether the system was simply rehashing statistical methods.
The teacher expressed concern that the system was rehashing statistical methods, rather than using a new form of intelligence, and that all the flaws, limits, and biases that are well-known in statistics were being swept under the rug by rebranding the system as artificial intelligence, thereby “blackboxing” the processes.
The teacher argued that blind faith in such an unchecked system of statistical governance may lead to the kinds of long-term problems that still plague the field of statistics.

Discussion Question #5

What is the danger of calling a system that deals in particulars “AI?”
What are the advantages?

Responses from Minerva High School and Hephaestats

The school argued that the data was pseudonymized and used for the legitimate purpose of improving educational outcomes.
Hephaestats resisted releasing its proprietary algorithms, arguing that the source code would be meaningless to the public.
However, Hephaestats provided tools to understand which data points were used to recommend actions.
Hephaestats acknowledged that their system was based on statistical methods but also used recent developments in machine learning.
They used the term "artificial intelligence" because it is more commonly used, while correcting for known limitations in the field.

Reflection & Discussion Questions

Privacy

Balancing individual privacy considerations against achieving legitimate social ends involves inevitable trade-offs.
The values embedded in systems reflect the priorities of the systems' designers.
In the Minerva case, the school board and Mr. Vulcani decided that some infringement of students’ privacy was a reasonable price to pay for a lower dropout rate.

Discussion Questions on Privacy

How should decisions about the appropriate balance between privacy and improving educational outcomes be made?
Who should be involved in this process and to what extent?
How does the issue of privacy change in the school setting? Would the appropriate balance between privacy and social welfare be different in a private school? In an office setting? Think of this question using both ethical and constitutional law frameworks.

Autonomy

Autonomy is an individual’s ability to make decisions for herself and act upon them.
AI systems like Hephaestats may compromise autonomy.
For teachers, the use of Hephaestats’ system could mean that their priorities and judgements are overridden.
For students, minors are not thought to have the same degree of autonomy as adults, either in theory or law. Treating minors as less autonomous can be good in terms of protecting more vulnerable members of society, but also raises some difficult questions about paternalism.

Discussion Questions on Autonomy

Should Hephaestats provide students with their risk profiles? Should students have a right of appeal? Should they be able to opt out of being assessed? Would it be possible to include them in decisions regarding the design and deployment of Hephaestats, and if so, how?
Hephaestats offered several options to address the student dropout rate at Minerva High School, but they mostly emphasized a “nudging” model. This meant that students typically did not receive explicit mandates or directives; rather, they were nudged along in certain directions through changes to the incentive structures (e.g. by making sugary foods less available). Nudging represents a softer form of influence, but it is influence nonetheless. AI systems designed to “nudge” are justified on the basis that they adjust choice architecture to help people make the right decisions. But the right decisions for whom? Who decides? Are nudges a better way of producing desired outcomes than more explicit exertions of power? Why or why not?

Consequentialism

Some people argue that certain actions are impermissible regardless of what good outcomes they might bring about; others believe that the ends may justify the means.
The Minerva administrators and their partner, Hephaestats, had both good ends and, they argued, appropriate means. But in complex AI systems, it may be quite challenging to even keep track of the various means in use. If all these means must be evaluated independently of the ends they’re used to bring about, it may be very difficult to evaluate the permissibility of different actions.
Furthermore, when AI is deployed to solve real world problems, each step of the implementation must be tracked as well. Considering the difficulty of assessing each of these steps in their entirety, school officials and Hephaestats preferred to focus on their noble end of reducing the student dropout rate.

Discussion Questions on Consequentialism

Even if nearly everyone felt the school dropout rate was a problem, not all stakeholders agreed with Hephaestats about the appropriateness of their means, namely, their use of student data without consent to produce un-auditable results. These dissenters might argue that the way Hephaestats went about reducing the dropout rate undermined its ultimate success in achieving this “noble” end. What would you say?
If we accept that all significant stakeholders ought to have a voice in determining the values they want their communities to promote, does it follow that they should be involved in decision-making about the means of achieving those ends as well? How would schools go about including them?

Rhetoric

The use of language is very important, especially in framing and describing new, developing technologies. Hephaestats chose to label and promote their IT solution as “artificial intelligence,” but they also could have labeled their approach as a matter of social science or statistics, instead.

Discussion Questions on Rhetoric

Was Hephaestats right to call its technology "AI"? If not, how should they have formulated their description of the system to the school board, students, parents, and teachers?
What are the implications of calling something "AI"? What kinds of political and social intentions does that decision reflect? Does the “AI” labeling evoke a kind of responsibility beyond what is typically expected from an IT system? If so, what could be done to mitigate these concerns?

AI Ethics Themes

Privacy
Autonomy
Consequentialism
Rhetoric