Week Six: Risk Assessment and Prediction of Criminal Behaviour
Introduction to Risk Assessment
We're diving into the world of risk assessment, specifically focusing on its use in predicting the likelihood of individuals committing new crimes or being charged with them. These tools are increasingly used by legal professionals, sparking considerable debate.
Key Terms and Roles
Recidivism: Refers to reoffending. Crucially, being assessed for recidivism doesn't automatically mean the individual is guilty of the original charges. Innocent individuals can be charged.
Defendant: The justice-involved individual. Being a defendant doesn't equate to guilt.
Defense Attorney: The lawyer representing the defendant.
Prosecutor: The lawyer bringing charges against the defendant.
Judge: Oversees the case and can decide on the verdict (depending on the case) or sentencing.
Magistrates: Volunteers who hear cases in community courts, involved in initial processing, making decisions on fines, fees, and whether someone should be detained or released while their case is ongoing. This is an important decision, considering "innocent until proven guilty."
Jail: Used for short-term sentences or detaining defendants during their pretrial period.
Prison: Typically used for long-term sentences.
What is a Risk Assessment?
A risk assessment is a tool or method to evaluate the likelihood of a justice-involved individual engaging in specific behaviors, such as committing a crime, being charged with a crime, or exhibiting mental health concerns like risk of suicide or harm to others. These tools predict future behaviors based on aggregated data and statistics, sometimes using algorithms to generate a risk score.
Legal professionals use these tools to make informed decisions, such as whether someone is a risk to society or themselves. The argument is that these tools help mitigate biases in judges' and magistrates' decisions. Decisions include whether to detain or release someone and whether intervention is needed.
Variety of Risk Assessment Instruments
Numerous risk assessment instruments exist and are continuously revised and updated. These instruments are widely used in the US and the UK, despite differences in the legal systems. The focus is on pretrial risk assessments, which occur between the time someone is charged and their case is disposed of (charges dropped, acquitted, or convicted).
During the pretrial stage, individuals might be detained, released with supervision (e.g., ankle monitors, meetings with supervisors, drug/alcohol treatment), or released without supervision or restrictions. The goal of pretrial risk assessment is to provide evidence-based estimation of risk to inform decisions about pretrial release.
These assessments are used by magistrates, judges, supervisory teams, and social workers.
Examples of Risk Assessment Instruments
COMPASS: Requires a social worker to fill out a form with the individual. It includes questions about the severity of current charges (violence, pending charges, felony vs. misdemeanor), criminal history (drug/sex offenses, parole violations, family criminal history, peer associations), substance abuse issues, length of time at current residence, social environment (gangs in the neighborhood), and education level. Contains over 100 questions.
Arnold New Criminal Arrest Risk Assessment Instrument: Specifically for violent criminal arrests. It assigns points based on specific details, and the total points indicate the person's risk level. Quick and easy to administer.
Virginia Pretrial Risk Assessment Instrument (Vipray): Used in Virginia and neighboring states. It has nine factors, with points assigned based on responses. Total points indicate risk level, ranging from 1 to 6.
These instruments vary in their approach, questions, and factors but aim to predict recidivism pretrial.
Performance of Risk Assessment Instruments & Signal Detection Theory
The performance can be described using signal detection theory. The total risk score produced by the instruments is analogous to the memory signal strength. The instrument should distinguish those who recidivate from those who don't.
Individuals with low scores show weak evidence of risky behavior, while those with maximum points show strong evidence. The distribution of risk scores for those who do not re-offend has a lower mean than the distribution for those who do recidivate.
The separation between these distributions is d-prime (d'$’). The goal is to adopt the instrument with the largest , which best separates recidivists from non-recidivists. Just like in eyewitness memory research, indicates performance.
Risk levels can be characterized as thresholds along the risk continuum. For example, a risk level of 5 might be considered low risk, 11 as moderate, and 16 as high. The amount of distribution within each bin (defined by these thresholds) gives a proportion correct.
This setup predicts a confidence accuracy plot, where the x-axis is risk level (low to high) and the y-axis is the chance of recidivism (0-100%). The model predicts a strong relationship between risk level and recidivism if the instrument is performing as designed.
Data on Risk Score Prediction of Recidivism
Criminologists have assessed the predictive validity of these instruments, particularly COMPASS, which uses over 100 factors. A crucial factor is whether d-primes between white and black defendants are small, indicating equivalent performance across groups.
COMPASS: is around 1 overall. A of 0 indicates chance performance. A of 1 is one standard deviation difference between the distributions, indicating many misclassifications. Calibration curves show the relationship between risk level and chance of recidivism, with the y-axis as chance of recidivism. While the relationship is upward sloping, it is not as steep as desired. The highest risk level corresponds to approximately 75% chance of recidivism. The COMPASS seems similarly predictive for both white and black defendants.
Arnold NCA: overall is 0.92. for white defendants is around 1, and for black defendants, it is 0.9, which is not a significant difference. However, for other races, decreases, but the sample size is low. There is a positive correlation with risk score and recidivism, but the highest level of risk has approximately a 50% chance of recidivism. There may be racial and sex biases for the "other" group, but there are limited data.
Vipray: is way less capable of distinguishing recidivists from non-recidivists. A similar is observed for white and black defendants, and there are no apparent racial biases. Calibration curves show most people have a risk score of 0, and the Vipray seems to predict risk, although error bars are large.
Fairness and Quality in Risk Assessment
Even if these instruments perform decently and show small differences across race and sex, it's important to question whether they are fair. Meta-analyses suggest these instruments predict rearrest rates and court appearances with fair to excellent validity and similar validity across racial groups.
When these tools are used, fewer people are held in pretrial detention, and if they are held, it is for a shorter time. However, racial disparities remain in the likelihood and length of pretrial detention after the instruments are implemented. This is primarily due to courts not using the instruments uniformly across defendants. One significant issue is transparency. Black box AI approaches may make legal professionals less likely to rely on risk assessments. Legal professionals should be able to audit and understand the factors that produced that risk.
Racist beliefs and lack of understanding can lead to disagreement with risk scores.
Companies produce risk assessment instruments with algorithms that are secret and cannot be shared with court actors. This practice is dangerous. COMPASS uses a black box algorithm, making it difficult to know how the factors are combined or weighted. On the other hand, the Arnold NCA and Vipray are glass box, displaying transparently how they are weighted and how they produce total risk scores.
Algorithms applied to solve problems in the legal system should be transparent and glass box to enable trust from legal professionals.
What To Do With People Deemed Low Risk
Detaining people pretrial can be disastrous, leading to job loss and loss of custody. Options include releasing them without supervision or community supervision.
Community supervision programs, such as access to education, housing assistance, and healthcare, require periodic check-ins, drug testing, and therapy. These programs are intended to mitigate or reduce recidivism.
One study examined a supervision program in North Carolina, comparing it to a similar group released without restrictions. The chance of recidivism between the two groups was extremely similar, indicating that the supervision program did not successfully reduce recidivism. All the resources allocated to the supervision programs did not necessarily help. People could've potentially just been released as opposed to participating or taking part in the program.
Despite the ability to predict risk, there's no guarantee for its successful mitigation of reducing risk. These supervision programs need substantial overhaul.