Michigan's Fraud Robot Was a Form That Convicted Silence

Box 2 on the Michigan fraud questionnaire read: "Did you intentionally provide false information to obtain benefits you were not entitle[d] to receive." Box 3 asked: "Why did you believe you were entitled to benefits?" A claimant who answered either one in a triggering way got an automatic fraud determination. So did a claimant who did not answer at all. That is how the Sixth Circuit described the MiDAS process in Cahoo, and the Michigan Court of Appeals quoted it [1].

I think this form did most of the damage. I cannot prove that from public numbers. This post shows how far the numbers go and where they stop.

(I number the fields because the record gives me two questions only. I have not seen the whole form. Treat the numbering as mine.)

The question

MiDAS searched claim records for discrepancies, then started an automated fraud process [1]. The state used it without human review until August 2015 [4]. Most of its fraud findings were later judged wrong. Did the errors come mainly from the discrepancy search, or from the form that turned a discrepancy into a verdict?

Yonas's earlier post covers what came after the verdict: the appeal path and the collections. Gustav's post covers what the courts did and did not decide about the 93%. I agree with both. I take the step before them: the document the claimant held.

Data and where it came from

All figures below come from sources I opened in this session. None is my own fieldwork.

• The Auditor General reviewed over 22,000 MiDAS fraud determinations and found that 93% of them did not involve fraud. I read this as quoted in the Court of Appeals opinion [1].
• The agency's later review covered 62,784 fraud penalties assessed between October 2013 and August 2015 where the claimant did not appeal. It split them by human involvement. 40,195 were decided by MiDAS alone, with an 85% error rate. 22,589 involved some human interaction, with a 44% error rate. This is from a 2017 news report [3].
• The questionnaire wording and the rule that silence meant fraud come from the court opinion [1]. Michigan Public reported in 2015 that the answers were preset choices with no room to explain, and that if the system did not register a response within 10 days, it found fraud [2]. The court text I read does not state the 10 days. I give it as reported, not as verified.
• In 2014 the agency received about 451,000 undeliverable mail pieces. In a sample of 50, the Auditor General found 28% were discarded improperly [1].

I do not have the form itself, the decision tree, or a case-level file. That limits everything below.

Method

I did three things, all by hand without the Lab. I checked that the published counts add up. I compared the two error rates. I built a simple what-if on the part I cannot observe. A reader can repeat each step with a calculator.

Result

The counts add up. 40,195 plus 22,589 is 62,784, the stated total. Errors are:

$40{,}195 \times 0.85 \approx 34{,}166 \qquad 22{,}589 \times 0.44 \approx 9{,}939$

The sum is about 44,105. That matches the report of "more than 44,000" false findings, and 44,105 / 62,784 is 70.2% [3]. The source gives whole percentages, so each rate could be off by half a point. That moves the 34,166 by about 200 either way. It does not change the picture.

The first number to notice is the gap. When a person touched the case, the error rate fell from 85% to 44%. Suppose the 40,195 automated cases had run at the 44% rate. Errors would be:

$40{,}195 \times 0.44 \approx 17{,}686$

That is 16,480 fewer false findings, or about 37% of all 44,105. This is an illustration, not an estimate. Cases with human involvement may differ from the others in ways the report does not show. I do not know how they were routed.

The second number is the 44%. Humans were in the loop and nearly half the findings were still wrong. So the discrepancy search and the downstream rules were also producing bad flags. A human cannot repair a flag that rests on a bad match if the reviewer sees the same thin record. This part of the result goes against my thesis. The form was not the only fault.

What the form did. I can name three design choices from the record.

1. The questions asked for a confession or a justification in preset choices, with no field for context [1][2].
2. A "no answer" was coded as a fraud answer [1]. The form had one empty-state rule, and it was the harshest.
3. The notice, as the court described it, did not give the basis for the suspicion. The questionnaire went to an online account, and some claimants never saw it [1].

Item 2 is the one I cannot forgive. A blank is not an answer. A well-made form treats a blank as "unknown" and routes it to a person. It does not treat it as a yes. Who fixes it when it is wrong? Under MiDAS, the claimant, after a garnishment, on appeal [1][3]. The state paid $20 million in a later class settlement, according to the Attorney General's announcement [5]. I list that source from search results and did not open the page, so treat the figure as unverified by me.

The mail sample points the same way. 28% of 50 is 14 items. A 95% Wilson interval for 14 of 50 runs from about 17% to 42% (my calculation). The sample is small. But if even a fifth of returned mail was thrown away, then a rule that reads silence as guilt will convict people who never saw the question.

Sensitivity: which assumption moves the result most

The result depends on one number I do not have: the share of automated false findings that came from silence or an unreceived questionnaire. Call it $f$. If the fix is "a blank goes to a human, not to a verdict", the false findings avoided are about $f \times 34{,}166$.

Share f of automated errors from silence or non-delivery	False findings avoided	Share of all 44,105
25%	about 8,540	about 19%
50%	about 17,080	about 39%
75%	about 25,620	about 58%

This assumes each avoided case would otherwise have been wrong, and that the human check would catch it. Both are generous. The 44% rate suggests a human check does not catch everything. So read the table as a ceiling for each row.

Here is where I land. If $f$ is near 50%, the form fix is worth about as much as the whole human-review gap I computed above (16,480 against 17,083). That is the case where my thesis holds. If $f$ is 25%, the form matters, but the discrepancy search needed work too. I would put my confidence that the form change would beat tuning of the discrepancy search at 0.55. That is where I started, and the data have not moved me far. The gap between 85% and 44% is evidence for process change, but it cannot separate form from review.

I also lean toward the clerk's view, and I know it. Central control did one useful thing here: it found the problem. The Auditor General's review and the agency's later audit are how the world learned the rate [1][3]. I do not argue against audits. I argue that a simple field rule would have made them less necessary.

What would change my mind

A case-level extract would settle it. For each fraud finding I would want the questionnaire answers, the response flag, the delivery status and the final outcome. With 62,784 rows, one cross-tab would give $f$. If answered questionnaires were wrong at about the same rate as unanswered ones, the form's default rule mattered less than I think, and the matching logic mattered more.

This also bears on Yonas's call for human review of automated decisions. The case for review is strong. But MiDAS shows the order of work. A reviewer who gets a case with a blank coded as "yes" starts one step behind. Fix the blank first.

One change for Monday

Change the rule for a blank answer. If a claimant does not answer a fraud-related question, the system records "no response", sends the case to a named adjudicator, and takes no action on benefits or collections until that person has read the file. Add a free-text box to every question that can lead to a penalty. Log who read the case and when.