Michigan's MiDAS: a 93 percent error rate, and an appeal path that let the state collect first

On December 3, 2014, Michigan's Unemployment Insurance Agency told Grant Bauserman he had committed fraud. It assessed $19,910 in penalties and interest. He protested on time, through the agency's own online appeal system. While that protest was still pending, on June 16, 2015, the agency intercepted his tax refund. On September 30, 2015, it reversed itself, declared its earlier notices "null and void," and returned the money [1].

That case is my test. Bauserman did everything the system asked of him. He used the appeal and he met the deadline. The state took his money anyway, 195 days after the finding and 107 days before anyone got around to deciding it was wrong. (Day counts are mine, from the dates in the Michigan Supreme Court's opinion.)

I started this piece with a thesis: MiDAS failed less because its fraud algorithm was inaccurate than because of the order of its appeal path. The numbers do not support the first half of that. The algorithm was wildly inaccurate. They do support a narrower claim, and it matters more for policy: the appeal design decided who paid for the inaccuracy. A human review before collection would have caught somewhere between about half and nearly all of the errors. Where in that range depends almost entirely on one assumption, which I name below.

Question

Two questions, kept apart on purpose. First, how wrong was MiDAS, measured by the state's own later reviews? Second, if Michigan had been bound by the rule I hold (no automated finding on public benefits becomes enforceable without human review within 30 days), how much of the error would that rule have stopped before money moved?

Data and where it came from

All figures are the state's own counts, as reported in court records and the press. I could not open the agency's August 11, 2017 release directly (the state server refused the request), so I rely on reporting that quotes it.

The process. The Sixth Circuit described the plaintiffs' allegations in Cahoo v. SAS Analytics (2019). From October 2013 to August 2015, "no human being took part in this process." MiDAS sent claimants a multiple-choice questionnaire and gave them ten days to answer. If a claimant answered any question in the affirmative, or did not answer, "MiDAS robo-adjudicated the fraud issue and automatically determined" that the claimant had knowingly misrepresented information. Claimants then had 30 days to appeal to an administrative law judge. According to the complaint, "the vast majority" did not learn of the finding until that window had closed. Penalties ran from $10,000 to more than $187,000 [2]. The questionnaires went to online accounts, often dormant ones, and did not say why fraud was suspected [1].

The penalty and the collection tools. Michigan's penalty was four times the principal, plus interest, the harshest in the country [3]. The agency collected through tax refund intercepts and administrative wage garnishment [4]. When MiDAS went live in 2013, the agency laid off about 400 full-time and part-time staff, roughly a third of its workforce [3].

Finality. One more feature closes the trap. A claimant who did not appeal within 30 days, "even if the claimant never received the decision to begin with," was treated as having a "final" fraud determination [5].

The error counts.

Population	Cases	Reversed	Rate	Source
Computer-only findings, Oct 1, 2013 to Aug 7, 2015 (2016 review)	22,427	20,965	93%	[6], [7], [8]
Algorithm-alone findings, Oct 2013 to Sep 2015 (2017 full review)	40,195	about 34,166	85%	[9]
Findings with some human involvement (2017 full review)	22,589	about 9,939	44%	[9]
Human-involved findings, interim count (May 2017)	about 14,470	6,872	47.5%	[10]

The reversed counts in rows two and three are my own calculation: the reported percentage times the case count, done by hand without the Lab. The two rows sum to about 44,105 false findings. Because the percentages are rounded to whole numbers, that sum is approximate. The interim denominator in row four is $6{,}872 / 0.475 \approx 14{,}470$, which is also my derivation.

Money moved in both directions. The agency's fraud collections reportedly rose from about $3 million to $69 million in just over a year [9]. By December 2016 it was refunding $5.4 million to 2,571 people [7]. In 2017 it committed to refund $20.8 million [4]. In the Bauserman class action, a $20 million settlement covering more than 3,000 people received final approval on January 30, 2024 [11].

Method

I set up two counterfactuals, because "human review" can mean two very different things.

Counterfactual A: ordinary front-line humans. Every algorithm-alone case gets the same kind of human involvement that the 22,589 "human-involved" cases got in 2013 to 2015, by the same depleted staff working from the same data. Those cases carried a 44 percent error rate, so I apply 44 percent to the 40,195 algorithm-alone cases.

Counterfactual B: review at the standard of the later reviews. Before any collection, someone reviews the file to the standard of the 2016 and 2017 retrospective reviews. By construction this catches whatever those reviews caught, because those reviews are my only measure of error.

The truth for a real 30-day rule lies between A and B. The decisive question is how good a reviewer would be under a deadline.

Result

Under A, false findings among algorithm-alone cases fall from

$$0.85 \times 40{,}195 \approx 34{,}166$$

to

$$0.44 \times 40{,}195 \approx 17{,}686.$$

That removes about 16,480 false findings, 48 percent of the algorithm's errors. Across all 62,784 cases, false findings fall from about 44,105 to about 27,625, a cut of 37 percent. Under B, nearly all of the roughly 44,000 false findings are caught before collection.

So my honest range is that a pre-collection human review would have stopped 48 percent to nearly 100 percent of MiDAS's false fraud findings before any refund was seized. My original thesis said "most." That holds under B and fails narrowly under A.

There is evidence that B is not a fantasy. Administrative law judges were reversing MiDAS findings as early as spring 2014. One wrote that "the fraud notice was issued in error due to the MiDAS system's inability to read fact-finding information" [6]. That error is visible to a human who opens the file. Many of these errors could be detected. The system was simply built so that nobody opened the file until the claimant forced it, and by then the money was gone.

Two more points stand apart from the error rate.

First, the timing harm does not depend on the error rate at all. Bauserman's case was reversed 301 days after the finding. A 30-day review clock would have forced a decision by January 2, 2015, 165 days before his refund was intercepted. Even under counterfactual A, where review is mediocre, the cases it misses are still contested before collection rather than after, provided the rule stays collection until review.

Second, a right that the claimant must trigger fails when notice fails. Michigan had an appeal. It had a 30-day window. Cahoo alleges that most people never learned of the finding inside that window [2]. A review that waits for a request is worth only as much as the notice that prompts it. Who can appeal? On paper, everyone. In practice, only people who checked a dormant web account within 30 days.

Sensitivity

Reviewer quality moves the result most. It is the whole gap between 48 percent and nearly all. Michigan's front-line human cases ran at 44 to 47.5 percent error [9][10]. That came from an agency that had just cut a third of its staff and was feeding people the same mangled inputs. A 30-day rule enforced by a rubber stamp delivers counterfactual A or worse.

Selection into the human-involved group. The 22,589 cases with human involvement were not a random sample. They may have been harder, or easier, than the algorithm-alone cases. If they were harder, A understates what ordinary review would have achieved on the algorithm's cases. I cannot sign this bias from the sources I read.

The review as ground truth. Every error rate here is the agency's later judgment of its earlier judgment. These are censuses of reviewed cases, not samples, so sampling intervals do not apply. The real uncertainty lies in the review standard. Suppose the later reviews were lenient and reversed some genuine fraud. Then the true error rate is lower than 85 to 93 percent, and the rule's benefit shrinks with it. I have no independent audit of the reviews. I treat the 85 and 93 percent figures as upper-leaning estimates and do not adjust them, because I have no number to adjust them with.

Which denominator. The widely quoted 93 percent covers 22,427 computer-only cases reviewed in 2016 [7]. The fuller 2017 count gives 85 percent on 40,195 [9]. My calculation uses 85 percent. Using 93 percent instead raises the algorithm's errors to about 37,381 and the share counterfactual A removes to 53 percent, which turns my "narrowly fails" into "narrowly holds." The thesis is that fragile at its weak end.

What the rule costs

I promised myself I would price a rule, so here is a rough figure. I could not find a sourced time per fraud adjudication, so the next numbers rest on my own assumption and should be read that way. Assume one careful review takes 2 hours. Then 40,195 algorithm-alone cases over about two years come to roughly 40,200 hours a year. At 1,800 productive hours per full-time employee, that is about 22 staff. At 6 hours per case it is about 67. Either figure is a fraction of the roughly 400 positions Michigan cut when MiDAS arrived [3]. It is also small beside the $20.8 million refunded [4] and the $20 million settlement [11], though I have not priced the staff in dollars because I have no sourced wage figure.

The rule has a real cost, and I will name it. In the 2016 review, 1,462 of the 22,427 computer-only findings survived (22,427 minus 20,965). Those people, and their counterparts in the larger set, would have kept their money for up to 30 more days, and a few might have become harder to collect from. That is the tradeoff: slower collection from the minority who committed fraud, in exchange for not seizing refunds from the majority who did not.

The rule

Name the decision maker. In Michigan from 2013 to 2015, the agency handed the decision right to software and kept the collection right for itself. The legislature's 2017 fix required fraud determinations to be made by people [4]. That addresses the first half of the problem. It says nothing about the order of events.

My rule: no tax intercept, garnishment or penalty may be enforced on a public-benefits fraud finding until a named human adjudicator has reviewed the file and signed it. If that review has not happened within 30 days of the finding, the finding lapses instead of becoming final. The review runs automatically. The claimant does not have to request it, because Michigan shows that a request-based right is only as good as the notice behind it.

The rule fails in a predictable way. Under a quota and a 30-day clock, tired reviewers will sign whatever the system produced, and the lapse provision will push the agency to clear files fast rather than well. That is counterfactual A, a 44 percent error rate with a signature on it. What would change my mind about the rule is evidence that signed reviews under deadline perform no better than the 2013 to 2015 human-involved adjudications. If that holds, the deadline buys only time, and the real fix is somewhere I have not looked: in the inputs, or in the staffing, rather than in the order of events.