AI and automated decisions, and the lives they reach. A documentary practice.
Long before today's AI, machines were already deciding whom to suspect: who is checked for fraud, who is offered extra care, who can rent a flat. Some of these systems use machine learning. Others are a few fixed rules: Robodebt, which wrongly took money from 381,000 people in Australia, divided a yearly income by 26.
Each work starts from one documented case, built from court rulings, inquiries and research, and drawn as two images. The Encounter Plate, like the eye here, shows the system. The Spectrogram shows what it did, to whom, and when someone stepped in.
SyRI: a fraud score for poor neighbourhoods
From 2014 the Dutch state allowed six public bodies, among them the tax office, the benefits agencies, municipalities and the immigration service, to pool their records on residents. A risk model that was never made public then flagged individual addresses for possible welfare fraud. It was used only in neighbourhoods officially labelled as problem districts, where incomes are low and many residents have a migration background. The people living there were not told they were being scored.
In February 2020 the District Court of The Hague stopped the system nationwide. It found that SyRI violated the right to private life: the model was too opaque to check, and profiling selected neighbourhoods could discriminate by income and origin. The ruling became a reference point for algorithm cases across Europe.
Why it opens the series. SyRI shows the whole cycle the work records in one case: a system, the people it targeted, the people who raised the alarm, a court that stopped it, and what remained afterwards. The data it produced stayed in government databases after the ruling.
- System
- Rule-based risk score
- Scored
- about 135,000 residents
- Stopped
- 5 February 2020, District Court of The Hague
- Worst harm
- Dignity, 8 of 10: people treated as suspects because of where they lived
Every mark in the images comes from the case record
The plate: what kind of system
- 1Pupil. Type of system. Lines, as here, mean fixed rules; a faint swirl, a statistical model; an empty pupil, machine learning (AI).
- 2Iris brightness and grain. How well documented the case is. Bright and fine, as here: courts and inquiries have examined it. Dim and grainy: the system is still secret.
- 3Red ring. How certain it is that the system caused the harm. Sharp, as here: proven in court. Blurred: estimated.
- 4Red point. The point after which harm could not be undone. The more people affected, the bigger it is.
The spectrogram: what it did, and when
- 1Seven rows. Seven kinds of harm: liberty, dignity, employment, family, housing, health, reputation.
- 2Brightness. How severe the harm was. Black is none, white is the maximum. Reputation, where the arrow points, is 7 out of 10.
- 3Solid white line. Someone stopped the system. Here, the court ruling of February 2020.
- 4Dotted texture. What remained after the ruling. The data the system produced stayed in agency databases.
Three kinds of pupil
More cases
Risk classification modelTens of thousands of families wrongly accused of benefit fraud, with nationality used as a risk factor. The Dutch government resigned over it in 2021.
Machine learning (AI)A hospital algorithm that ranked Black patients as healthier than they were, so fewer received extra care. None of the patients were ever told.
Fixed rulesA spreadsheet formula turned income averages into debts and sent them to 433,000 people. A Royal Commission found it unlawful from the start.
Essays
Major General (retd.) Juan A. Moliner's account of human control over autonomous weapons, read alongside three civilian cases. In each case the human who was supposed to be able to intervene took years to do so. The essay argues that "on the loop" control is a matter of time and authority, and that responsibility has to follow the decision back to its designers.
Public attention to harmful automated systems usually ends at the moment of recognition, when a court, an inquiry or a regulator finds that the system was wrong. Using the five forms of reparation in the UN Basic Principles on the right to a remedy as a frame, the essay follows six cases past that moment. Cessation is common, compensation partial, rehabilitation rare, and guarantees of non-repetition have repeatedly failed.