The word in the search bar
Most people who look for information about automated decisions now type “AI”. The debate, the funding and the regulation have followed the word. This creates a problem for anyone who documents what automated systems have actually done to people, because several of the most damaging cases of the past decade involved no artificial intelligence in any technical sense.
The clearest example is Australia’s Robodebt scheme. It took each welfare recipient’s annual income as reported to the tax office, divided it by 26, and compared the result with the income the person had declared for each fortnight. Any difference became a debt. About 433,000 people received such debts between 2016 and 2019, and A$751 million was wrongly recovered from 381,000 of them. The Royal Commission that examined the scheme found it unlawful from the outset (Royal Commission into the Robodebt Scheme 2023). The technical core of the scheme was one division and one comparison.
What the AI Act covers
The EU’s Artificial Intelligence Act defines an AI system as a machine-based system that “infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions” (Regulation (EU) 2024/1689, Article 3(1)). Recital 12 adds that the definition should not cover “systems that are based on the rules defined solely by natural persons to automatically execute operations”. The Commission’s guidelines on the definition, published in February 2025, go further and name categories that fall outside it even when they perform some inference: basic data processing, systems based on classical heuristics, simple prediction systems and certain optimisation methods (European Commission 2025).
There are good reasons for drawing the line this way. A law that covered every spreadsheet would be impossible to enforce, and the specific risks of learning systems, such as opacity, drift and unpredictable generalisation, deserve specific rules. The consequence is still worth stating plainly. A system of the Robodebt type, built in an EU member state today, would most likely fall outside the AI Act. Its high-risk obligations for public benefit systems, including risk management, human oversight and the right to an explanation of individual decisions, would not apply.
Harm does not follow technique
The first series of my work documents six cases. Two are clearly not machine learning: Robodebt, and the French family-benefits fund’s risk score, which is a logistic regression on about forty variables whose code the fund has now published. One is clearly machine learning: the commercial healthcare algorithm audited by Obermeyer and colleagues (2019), which used past healthcare costs as a stand-in for medical need and so ranked Black patients as healthier than equally sick white patients. The others are harder to classify.
Coding the severity of harm in each case on seven dimensions, and never adding them up, gives a result that does not sort by technique. The machine-learning case is serious on health and dignity. Robodebt records harm on all seven dimensions, and its health record includes deaths that families linked to the scheme before the Royal Commission.
What the severe cases share is something else. Each placed an automated output inside a process that treated it as a finding rather than a lead. Robodebt reversed the burden of proof, so that recipients had to disprove a debt the state had never established. The Dutch childcare-benefits process demanded full repayment of years of benefits from families flagged as risks. In both, the damage was done by the administrative setting the output entered, and by the decision to act on it without independent checks. Alston (2019), reporting to the UN General Assembly on the “digital welfare state”, described the same pattern across many countries: automation introduced as a cost-saving measure into systems already organised around suspicion of the poor.
When the technique is unknown
For two of the six cases the question “is this AI?” cannot be answered with confidence from the public record.
SyRI, the Dutch risk-scoring system stopped by the District Court of The Hague in 2020, combined records from six public bodies and flagged addresses according to risk indicators that were never published. The court held that, because neither the model nor its indicators were disclosed, the system could not be checked, by the court or by the people it scored (Rechtbank Den Haag 2020). Commentators have since described it both as a rule-based matching system and as a machine-learning model. Amnesty International (2021) described the risk classification used in the Dutch childcare-benefits affair as including a self-learning mechanism; official investigations focused on its use of nationality as a risk factor. My own coding of these two cases records the classification that the case record supports, and the uncertainty around it.
This uncertainty is itself a finding. A definition that turns on technique gives the operator of a secret system the first word on whether the law applies. The people being scored cannot see the model, and courts and regulators have often learned what kind of system they were dealing with only years later, if at all.
The older rule worked better
European data protection law has had a technique-neutral rule since 1995. Article 22 of the GDPR gives a person the right not to be subject to a decision “based solely on automated processing” that has legal or similarly significant effects. It does not ask how the processing infers anything.
In December 2023 the Court of Justice of the EU applied that rule to SCHUFA, the German credit agency that scores about 68 million people. The Court held that where a lender draws strongly on a score to decide on a contract, producing the score is itself an automated decision within Article 22 (Court of Justice 2023, C-634/21). The ruling did not depend on whether SCHUFA’s model is statistical or learned, and it would have applied equally to a fixed formula. It is the clearest legal success in my series, and it came from the older law.
The comparison has limits. Article 22 protects individuals one decision at a time and offers little against a system that harms a population through many small decisions; the SyRI case, in which residents were never told they were being scored, was decided on the right to private life instead. The AI Act, by contrast, regulates systems before they are deployed. The two instruments are designed to work together. The point is narrower: the instrument that asks what the decision does has reached cases that the instrument that asks how the system works would not.
What to call it
None of this argues for dropping the word “AI”. It is the word people use, and a public record that refuses it will not be found. It argues for keeping a distinction inside the record. I label every case with the type of system as far as the evidence allows: machine learning, statistical model, fixed rules. The distinction matters for two reasons. It shows that the harms now discussed under the heading of AI have a longer history than the technology. And it locates responsibility. A rule written by a civil servant has an author in a way that a learned weight does not, and the Robodebt record shows that even an identifiable author can take a decade and a Royal Commission to reach.
References
- Alston, P. (2019). Report of the Special Rapporteur on extreme poverty and human rights (digital welfare state). UN General Assembly, A/74/493, 11 October 2019.
- Amnesty International (2021). Xenophobic Machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal. EUR 35/4686/2021.
- Court of Justice of the European Union (2023). Case C-634/21, OQ v Land Hessen (SCHUFA Holding), judgment of 7 December 2023.
- European Commission (2025). Guidelines on the definition of an artificial intelligence system established by Regulation (EU) 2024/1689. February 2025.
- Obermeyer, Z., Powers, B., Vogeli, C., and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
- Rechtbank Den Haag (2020). Judgment of 5 February 2020, ECLI:NL:RBDHA:2020:865 (English translation ECLI:NL:RBDHA:2020:1878).
- Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22.
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 3(1) and Recital 12.
- Royal Commission into the Robodebt Scheme (2023). Report. Canberra, 7 July 2023.