This Algorithm Could Ruin Your Life


It was October 2021, and Imane, a 44-year-old mother of three, was still in pain from the abdominal surgery she had undergone a few weeks earlier. She certainly did not want to be where she was: sitting in a small cubicle in a building near the center of Rotterdam, while two investigators interrogated her. But she had to prove her innocence or risk losing the money she used to pay rent and buy food.

Imane emigrated to the Netherlands from Morocco with her parents when she was a child. She started receiving benefits as an adult, due to health issues, after divorcing her husband. Since then, she has struggled to get by using welfare payments and sporadic cleaning jobs. Imane says she would do anything to leave the welfare system, but chronic back pain and dizziness make it hard to find and keep work.

In 2019, after her health problems forced her to leave a cleaning job, Imane drew the attention of Rotterdam’s fraud investigators for the first time. She was questioned and lost her benefits for a month. “I could only pay rent,” she says. She recalls the stress of borrowing food from neighbors and asking her 16-year-old son, who was still in school, to take on a job to help pay other bills. 


Now, two years later, she was under suspicion again. In the days before that meeting at the Rotterdam social services department, Imane had meticulously prepared documents: her rental contract, copies of her Dutch and Moroccan passports, and months of bank statements. With no printer at home, she had visited the library to print them. 

In the cramped office she watched as the investigators thumbed through the stack of paperwork. One of them, a man, spoke loudly, she says, and she felt ashamed as his accusations echoed outside the thin cubicle walls. They told her she had brought the wrong bank statements and pressured her to log in to her account in front of them. After she refused, they suspended her benefits until she sent the correct statements two days later. She was relieved, but also afraid. “The atmosphere at the meetings with the municipality is terrible,” she says. The ordeal, she adds, has taken its toll. “It took me two years to recover from this. I was destroyed mentally.”


Imane, who asked that her real name not be used for fear of repercussions from city officials, isn’t alone. Every year, thousands of people across Rotterdam are investigated by welfare fraud officers, who search for individuals abusing the system. Since 2017, the city has been using a machine learning algorithm, trained on 12,707 previous investigations, to help it determine whether individuals are likely to commit welfare fraud. 

The machine learning algorithm generates a risk score for each of Rotterdam’s roughly 30,000 welfare recipients, and city officials consider these results when deciding whom to investigate. Imane’s background and personal history meant the system ranked her as “high risk.” But the process by which she was flagged is part of a project beset by ethical issues and technical challenges. In 2021, the city paused its use of the risk-scoring model after external government-backed auditors found that it wasn’t possible for citizens to tell if they had been flagged by the algorithm and some of the data it used risked producing biased outputs. 

In response to an investigation by Lighthouse Reports and WIRED, Rotterdam handed over extensive details about its system. These include its machine learning model, training data, and user operation manuals. The disclosures provide an unprecedented view into the inner workings of a system that has been used to classify and rank tens of thousands of people.

With this data, we were able to reconstruct Rotterdam’s welfare algorithm and see how it scores people. Doing so revealed that certain characteristics—being a parent, a woman, young, not fluent in Dutch, or struggling to find work—increase someone’s risk score. The algorithm classes single mothers like Imane as especially high risk. Experts who reviewed our findings expressed serious concerns that the system may have discriminated against people