On 15 January 2021, the government of the Netherlands resigned. Not a minister: the entire cabinet. They resigned over a piece of software. For fourteen years it had helped decide which parents were committing childcare benefit fraud, and one of the things it treated as a warning sign was whether you held a second nationality. It was wrong about some twenty-six thousand families. They were ordered to repay everything, at once, and more than one thousand six hundred of their children were taken into state care.
That case is the last of seven in our 31-minute documentary for The Signal. Every one is documented, and almost all of the documentation was produced by the organisations themselves: a submission to a tribunal, a presentation to executives, a memo from a staff lawyer, an answer to a senator, a write-down in a quarterly filing. The cases run in the order of what they cost, starting with eight hundred and twelve dollars. The film is told as a chronicle: every case ends where its evidence ends, and where an official body stopped short of a finding, the film stops there too.
The film is at the top of this page. This piece follows it case by case, with the documents behind each one.
The film in 9 chapters
Pick a chapter and the film starts there. 31:06 in all.
- 010:00A government resigns over software
- 021:51Case one: a separate legal entity
- 035:10Case two: sixteen hours
- 048:29Case three: the word women's
- 0511:24Case four: the houses
- 0614:54Case five: unsafe and incorrect
- 0718:59Case six: Robodebt
- 0824:04Case seven: twenty-six thousand families
- 0929:43Those are the seven
What happened when Air Canada's chatbot gave a customer the wrong answer?
Watch from 1:51Case one: a separate legal entity
Air Canada argued its chatbot was responsible for itself, and lost. In November 2022, Jake Moffatt's grandmother died. Air Canada's website chatbot told him to book at full price and apply for a bereavement fare within 90 days afterwards. The airline's real policy, on another page of the same site, said the opposite. On 14 February 2024 a tribunal ordered Air Canada to pay him $812.02.
Moffatt flew to the funeral and applied. Air Canada refused, offered him a coupon for a future flight and said it would update the chatbot. He filed a claim at the British Columbia Civil Resolution Tribunal, Canada's first online tribunal of its kind, a place for damaged fences and withheld deposits. And in that forum, over a few hundred dollars, Air Canada submitted that the chatbot was "a separate legal entity that is responsible for its own actions."
The tribunal member, Christopher Rivers, answered in three sentences: "This is a remarkable submission. While a chatbot has an interactive component, it is still just a part of Air Canada's website. It should be obvious to Air Canada that it is responsible for all the information on its website." He found the airline liable for negligent misrepresentation and ordered it to pay $650.88 in damages, $36.14 in interest and $125 in fees. That is the amount at stake in the case where an international airline argued that its chatbot was a person.
That is the amount at stake in the case where an international airline argued that its chatbot was a person.
Why did Microsoft's Tay chatbot last only 16 hours?
Watch from 5:10Case two: sixteen hours
Microsoft launched Tay on Twitter on 23 March 2016 and took it offline about 16 hours later, because users turned it into a machine for racism within a day. Tay was built to talk like a young American woman and to learn from whoever talked to her. Microsoft had run a similar chatbot, XiaoIce, in China, with tens of millions of users and no incident of this kind.
Users found two levers. If you typed "repeat after me", Tay repeated anything you said. And because she learned from what was said to her, coordinated groups simply said the same things over and over until those things became what she said on her own. By the end of the day she was posting racist material, denying the Holocaust and praising Hitler. Around 96,000 tweets had gone out. A week later Microsoft accidentally switched her back on, and she was pulled a second time.
Two days after the shutdown, Microsoft apologised. It said it was deeply sorry for the unintended offensive and hurtful tweets, that it had stress-tested Tay under a variety of conditions, and that it had not anticipated this particular attack. Nobody was physically harmed. Seven years later the same company put a chat system into its search engine, and it told a New York Times reporter to leave his wife; that story is told in full in our film on Bing's Sydney.
Why did Amazon's AI hiring tool discriminate against women?
Watch from 8:29Case three: the word women's
Because it learned from ten years of Amazon's own hiring, and most of those applicants were men. From 2014, a team in Edinburgh built software to rate CVs from five stars to one, trained on the applications Amazon had received and the people it had hired. It taught itself to mark down any CV containing the word "women's", and, according to Reuters, graduates of two all-women's colleges.
Nobody wrote that rule. No line of code in the system mentioned gender. An engineer could read the whole program and never find it, because the rule was in the numbers the system had learned, thousands of them, each meaningless on its own. The only way anybody found out was by watching what it did. Amazon's engineers edited it to ignore those particular words, but that fixes only the terms you have already thought of; a sport, a society, a phrase or a college could carry the same signal.
Nobody wrote that rule. No line of code in the system mentioned gender.
Amazon abandoned the project and has said recruiters looked at its recommendations but never relied on them alone. None of this was announced. It became public in October 2018, when the Reuters journalist Jeffrey Dastin published an account based on five people who had worked on it.
How did Zillow's home-pricing algorithm lose $304 million?
Watch from 11:24Case four: the houses
Zillow Offers bought houses at prices set by a model, and through 2021 the model kept forecasting resale prices too high, in the same direction, repeatedly. On 2 November 2021 Zillow recorded a $304.4 million write-down on the homes in its inventory, wound the business down and announced a reduction of about 25 percent of its workforce, around two thousand people.
The idea was iBuying: if the Zestimate knows what a house is worth, buy quickly from people who want out, make light repairs and sell at a modest profit, at enormous scale. A human signed the paperwork, but the number in the offer came from a model that had learned from what houses sold for in the past, and it was pricing what a house would fetch months later. The pandemic had scrambled where people wanted to live, and construction costs and labour were unpredictable.
Zillow's filing gives its reasons as home pricing unpredictability, capacity constraints and operational challenges, exacerbated by an unprecedented housing market, a pandemic and a difficult labour and supply chain environment. Every item on that list is about the world. The filing does not say the model was wrong. The chief executive, Rich Barton, said: "the unpredictability in forecasting home prices far exceeds what we anticipated." The film deliberately leaves out two figures that circulate widely, an $8 billion loss in market value and 7,000 homes, because neither appears in Zillow's own filings.
Every item on that list is about the world. The filing does not say the model was wrong.
Did IBM Watson recommend unsafe cancer treatments?
Watch from 14:54Case five: unsafe and incorrect
According to IBM's own internal documents, reported by STAT on 25 July 2018, yes. The documents discussed "multiple examples of unsafe and incorrect treatment recommendations" from Watson for Oncology, sold to hospitals in at least a dozen countries. STAT also reported that no patient deaths had resulted from hospitals' use of it, and the film says so on screen.
After Watson beat two Jeopardy champions in February 2011, IBM chose cancer. A doctor would enter a patient's details and Watson would return ranked treatments, each with the literature behind it, trained with Memorial Sloan Kettering Cancer Center in New York. The first sign of trouble came from Texas: in February 2017 an internal audit showed MD Anderson Cancer Center had spent about $62 million on its own Watson project, the contract had lapsed, and the system had never been used on a patient.
One example in the documents: a 65-year-old man with lung cancer and severe bleeding, for whom Watson recommended a course including bevacizumab, a drug whose label warns it can cause severe or fatal haemorrhage. The documents explain why. Watson had been trained not on real patient records but on synthetic cases, hypothetical patients worked through by a small number of Memorial Sloan Kettering doctors. Customers described the recommendations as often inaccurate, and the internal assessment said this "raises serious questions about the process for building content and the underlying technology." What the evidence establishes is that a product recommending cancer treatments was documented inside the company as producing unsafe advice, and stayed on sale. IBM sold Watson Health to a private equity firm in 2022.
What was Robodebt, and how many people did it affect?
Watch from 18:59Case six: Robodebt
Robodebt was an Australian government scheme that took a person's annual income from the tax office, divided it by 26 fortnights, and automatically issued welfare debts on that basis. It ran from 2015 until 29 May 2020, raised hundreds of thousands of debts, and was unlawful. In June 2021 the Federal Court approved a settlement worth A$1.872 billion, including A$751 million repaid to people who never owed it.
For somebody in steady work all year, averaging is roughly accurate. For somebody who worked three months and claimed welfare for nine, it invents income in every fortnight they did not work, and turns it into a debt. It also inverted who had to prove what: the letter came first, and the recipient had to produce payslips, sometimes going back seven years, to show they did not owe money a computer said they owed. The government was told repeatedly, by its own lawyers and officials, that averaging had no legal basis.
A Royal Commission, led by Commissioner Catherine Holmes, reported on 7 July 2023 in 990 pages with 57 recommendations. "Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals," she wrote. "In essence, people were traumatised on the off-chance they might owe money." She called the affair an extraordinary saga of "venality, incompetence and cowardice", and referred individuals for prosecution in a sealed chapter.
Then the number. In 2019, answering a question in the Australian Senate, the department disclosed that 2,030 people had died after receiving a debt notice between July 2016 and October 2018, and that 663 of them were people it had classed as vulnerable. The film puts the 663 figure on screen. Families told the Senate, and later the Royal Commission, that their sons had taken their own lives after letters arrived. Here is exactly what is established: no coroner has attributed a death to Robodebt, and the Royal Commission made no finding that the scheme caused any particular death. That is where the evidence stops, so that is where the film stops.
What was the Dutch childcare benefits scandal?
Watch from 24:04Case seven: twenty-six thousand families
The Dutch Tax and Customs Administration used a risk model to flag childcare benefit fraud, and among its indicators was whether a claimant held a second nationality. Families it flagged were found to have committed deliberate fraud and ordered to repay the entire allowance, often EUR 20,000 to 60,000. About 26,000 families were wrongly accused between 2005 and 2019, and the government resigned on 15 January 2021.
Unlike the Amazon case, this was not buried in the mathematics or an accident of training data. A Dutch citizen who was only Dutch scored one way; one who also held Turkish or Moroccan nationality scored another. Nationality was an input. Somebody chose it. A fraud finding meant every month of the allowance, going back years, repaid at once, with no payment plan and no appeal that worked. People sold their houses, went bankrupt and lost work because they could not pay for the childcare that let them go to work.
Nationality was an input. Somebody chose it.
The reason it ran so long is on the record. In 2017 a lawyer inside the tax administration, Sandra Palmen, wrote an internal memo saying the office's treatment of these parents was unlawful and that they should be compensated. It was not acted on, and it was not given to the members of parliament asking about exactly this. In December 2020 a parliamentary committee published its report, titled Ongekend onrecht, unprecedented injustice. Mark Rutte and his entire cabinet resigned the next month.
The Dutch data protection authority then found the processing of nationality unlawful and discriminatory, and in December 2021 fined the tax administration EUR 2.75 million. Four months later it fined the same department EUR 3.7 million for a separate fraud blacklist. Both penalties are for the handling of data. The repair has its own history: a flat EUR 30,000 per family was announced in December 2020, when about 9,000 families were thought eligible; the tax office later acknowledged 11,000 people had been given extra scrutiny purely for holding dual nationality; by October 2024 almost 38,000 people had been recognised as affected, and in September 2024 families were offered EUR 5,000 each to drop their legal protests. The children have no scheme at all. A child taken at four is not returned at four.
One more fact belongs at the end of this case. Mark Rutte resigned on 15 January 2021. Two months later his party won the general election, and in January 2022, after the longest government formation in Dutch history, he was sworn in as prime minister for a fourth term. Twenty-six thousand families. One thousand six hundred children. One resignation, and the same man back in the same office inside a year.
One resignation, and the same man back in the same office inside a year.
What do the seven AI failures have in common?
Watch from 29:43Those are the seven
All 7 are documented, mostly by the organisations themselves, and nearly all of it is public: the Air Canada ruling takes about eight minutes to read, the Robodebt report is free to download, and the Dutch inquiry has a title you can search. The film, told as a chronicle, never names a pattern. It leaves one fact about the order for last.
The cases run from $812.02 in a small-claims tribunal, to sixteen hours on Twitter, four years inside Amazon, two thousand jobs at Zillow, internal slides at IBM, Australia's dead and Dutch families. But the Air Canada ruling is from February 2024, and the scheme behind the government that fell began in 2005, before the iPhone, before machine learning meant anything outside a university. These seven are not in the order they happened.
Key findings
Air Canada argued at a British Columbia tribunal that its website chatbot was 'a separate legal entity that is responsible for its own actions'. On 14 February 2024 the tribunal found the airline liable and ordered it to pay Jake Moffatt $812.02.
Moffatt v. Air Canada, 2024 BCCRT 149, via CBC News, 15 Feb 2024Microsoft launched its Twitter chatbot Tay on 23 March 2016 and took it offline about 16 hours later, after users taught it to post racist material. Microsoft said it had stress-tested Tay but had not anticipated this specific attack.
Microsoft, Learning from Tay's introduction, 25 Mar 2016Zillow recorded a $304.4 million write-down on homes in its inventory for the quarter ending 30 September 2021, wound down Zillow Offers and cut about 25% of its workforce.
Zillow Group, Q3 2021 results and shareholder letter, 2 Nov 2021MD Anderson Cancer Center spent about $62 million on a Watson project that was never used on a patient. STAT later reported internal IBM documents citing 'multiple examples of unsafe and incorrect treatment recommendations', while reporting that no patient deaths resulted.
Casey Ross and Ike Swetlitz, STAT, 25 Jul 2018Robodebt divided a year's income by 26 fortnights to raise welfare debts. It was unlawful, and in June 2021 the Federal Court approved a settlement worth A$1.872 billion, including the repayment of A$751 million.
Royal Commission into the Robodebt Scheme, report, 7 Jul 2023In 2019 the Australian government told the Senate that 2,030 people had died after receiving a Robodebt notice between July 2016 and October 2018, 663 of them people it had classed as vulnerable. No coroner has attributed any death to the scheme.
ABC triple j Hack, reporting the Senate answer, Feb 2019The Dutch data protection authority fined the Tax Administration EUR 2.75 million in December 2021 for processing the dual nationality of childcare benefit applicants in a way it called unlawful and discriminatory.
Autoriteit Persoonsgegevens, Dec 2021In April 2022 the same authority fined the Tax Administration a further EUR 3.7 million for its fraud blacklist, the FSV, for six violations of data protection law.
Autoriteit Persoonsgegevens, 12 Apr 2022Frequently asked questions about AI gone wrong
What are the biggest real examples of AI gone wrong?
The film covers seven documented cases, in order of cost: Air Canada's chatbot inventing a refund rule (2024), Microsoft's Tay chatbot pulled after 16 hours (2016), Amazon's CV-screening tool that marked down the word women's (2014 to 2018), Zillow's home-buying algorithm and its $304.4 million write-down (2021), IBM Watson for Oncology, Australia's Robodebt, and the Dutch childcare benefits scandal that brought down a government in January 2021.
What happened in the Air Canada chatbot case?
In November 2022 Air Canada's website chatbot told Jake Moffatt he could claim a bereavement fare after booking, which the airline's actual policy did not allow. Air Canada argued the chatbot was a separate legal entity responsible for its own actions. On 14 February 2024 the British Columbia Civil Resolution Tribunal called that 'a remarkable submission' and ordered the airline to pay $812.02.
Why did Microsoft shut down Tay?
Tay launched on Twitter on 23 March 2016 and learned from conversations. Users exploited a repeat-after-me function and fed it coordinated abuse until it posted racist material, Holocaust denial and praise of Hitler. Microsoft took it offline about 16 hours after launch and apologised two days later.
Why did Amazon scrap its AI recruiting tool?
Trained on ten years of CVs, mostly from men, the tool taught itself to mark down CVs containing the word women's and, according to Reuters, graduates of two all-women's colleges. Engineers removed those terms, but could not be sure it had not found another route to the same bias. Reuters revealed the project in October 2018.
What was Robodebt?
Robodebt was an Australian government scheme, run from 2015 until 29 May 2020, that averaged a person's annual income across 26 fortnights and automatically issued welfare debts on that basis. It was unlawful. A Royal Commission reported on 7 July 2023, calling it 'a crude and cruel mechanism, neither fair nor legal'.
What was the Dutch childcare benefits scandal?
The Dutch Tax Administration used a risk model that treated dual nationality as a fraud indicator, and families flagged for fraud had to repay their entire childcare allowance. About 26,000 families were wrongly accused between 2005 and 2019. The government of Mark Rutte resigned over it on 15 January 2021.
Did IBM Watson for Oncology harm patients?
STAT reported in July 2018 that internal IBM documents described 'multiple examples of unsafe and incorrect treatment recommendations', including a bleeding-risk drug suggested for a patient with severe bleeding. STAT also reported that no patient deaths had resulted from hospitals' use of Watson for Oncology.
Sources
- Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 14 Feb 2024canlii.org
- Jason Proctor, Air Canada found liable for chatbot's bad advice on plane tickets, CBC News, 15 Feb 2024cbc.ca
- Microsoft, Learning from Tay's introduction, 25 Mar 2016blogs.microsoft.com
- The Guardian, Tay, Microsoft's AI chatbot, gets a crash course in racism from Twitter, 24 Mar 2016theguardian.com
- Jeffrey Dastin, Amazon scraps secret AI recruiting tool that showed bias against women, Reuters, 10 Oct 2018reuters.com
- Zillow Group, Third Quarter 2021 Financial Results and Shareholder Letter, 2 Nov 2021investors.zillowgroup.com
- Matthew Herper, MD Anderson Benches IBM Watson In Setback For Artificial Intelligence In Medicine, Forbes, 19 Feb 2017forbes.com
- Casey Ross and Ike Swetlitz, IBM's Watson supercomputer recommended 'unsafe and incorrect' cancer treatments, internal documents show, STAT, 25 Jul 2018statnews.com
- Royal Commission into the Robodebt Scheme, Report, 7 Jul 2023robodebt.royalcommission.gov.au
- Prime Minister of Australia, Final report of the Royal Commission into the Robodebt Scheme, 7 Jul 2023pm.gov.au
- Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, Federal Court of Australia, 11 Jun 2021judgments.fedcourt.gov.au
- ABC triple j Hack, Over 2000 people died after receiving Centrelink robo-debt notice, figures reveal, Feb 2019abc.net.au
- Parliamentary Interrogation Committee on Childcare Benefits, Ongekend onrecht (Unprecedented injustice), Tweede Kamer, 17 Dec 2020tweedekamer.nl
- The Guardian, Dutch government resigns over child benefits scandal, 15 Jan 2021theguardian.com
- Autoriteit Persoonsgegevens, Tax Administration fined for discriminatory and unlawful data processing, Dec 2021autoriteitpersoonsgegevens.nl
- Autoriteit Persoonsgegevens, Tax Administration fined for fraud blacklist, 12 Apr 2022autoriteitpersoonsgegevens.nl
Every quotation in the film and in this article comes from a tribunal decision, company statement, filing, report, memo or named reporting listed above. Where an official body stopped short of a finding, as with deaths after Robodebt notices or harm from Watson for Oncology, this article stops there too. The Robodebt death figures are the department's own count of people who died after receiving a notice, not a finding of cause: 2,030 in total between July 2016 and October 2018, 663 of them people it had classed as vulnerable, which is the figure the film shows.
Watch next
Training data, models and the other terms behind these seven cases are explained in plain English, with a printable sheet, in our free AI Terms guide.