The promise and the problem with "Fact Checking" (UPDATED)

This morning, Glenn Kessler of the Washington Post discussed former HHS Secretary Kathleen Sebelius's recent quote, speaking at a rally, that "There are now 22 million people with affordable coverage thanks to the Affordable Care Act, and that’s a big deal and that number will grow.”

As regular visitors to this site know, my own "estimated total, all sources" number shown on The Graph currently ranges between 24 - 29 million people...but is in turn broken down into different types of enrollments, with different caveats, disclaimers and so forth depending on what criteria you're trying to measure.

In the case of Sebelius's quote, Kessler is attempting to parse 3 specific points: The number (22 million); the description ("affordable coverage") and the credit ("thanks to the ACA").

What's interesting about Kessler's piece is that as he breaks out the numbers, he acknowledges that while Sebelius overstates some of them, she also understated others...and she actually left out a couple more, although they're fairly minor ones:

‘8 million in market’

Eight million refers to the number of people who selected a plan in the ACA exchanges. At this point, we don’t know how many have actually paid their first month’s premiums, but most analysts are assuming a rate of about 90 percent, which would be about 7.2 million. While the sign-up period has ended, people who have a change of life situation, such as a new child or a marriage, can still buy a new plan, so the year-end number is still a moving target. Charles Gaba, who closely tracks these numbers at, believes the number of paid enrollees is now about 7.7 million.

Bingo; Kessler nails this on the head--Sebelius is referring to the 8.02 million who enrolled as of 4/19. The irony here is that even though she included the roughly 10% of these who didn't end up paying their first month's premium, she's still correct because by my count there've been around another 600K who have enrolled since 4/19...of whom the vast majority have also paid their premiums, meaning that her "8 million enrolled in the market" is (roughly) correct anyway, if you're willing to round the number up. In other words, she's correct, but for the wrong reason. Whether overstating the number by 260,000 people should be considered a "rounding error" depends on your POV.

Sebelius: 8 million; Me: 7.7 million

‘6 million (and counting) additional Medicaid folks’

Here, Sebelius appears to be relying on an April report from HHS, which counted about 5 million new enrollees and about 1 million transferred from state programs for the poor. There’s also another 2 million or so people who were previously eligible for Medicaid but never signed up; they came out of the “woodwork” because of the publicity about Obamacare.

Experts differ on whether they should be included or not, but Sebelius appears to have taken a conservative approach with this element of her calculation. There is no sign-up deadline for Medicaid, so that number has likely grown since April as well.

Again, a fair assessment. I actually have the Medicaid/CHIP range down as being much higher: Between 7 million and 9.4 million. There are two main reasons for this: First, some of the state data I have includes determinations (ie, people who have been determined to be eligible for Medicaid/CHIP), while other state data is strictly for people actually enrolled in the programs. The Medicaid/CHIP side of things is much fuzzier--some states have backlogs of people who have been approved but not actually enrolled yet, and so on. The other reason for the 1.0 - 3.4 million difference is that, as Kessler noted, that "6 million" figure was as of 4/19; since Medicaid is year-round, these numbers have gone up considerably in the past 2 1/2 months.

Sebelius: 14 million; Me: Between 14.7 - 17.1 million

‘3 million young adults’

This refers to the number of adults under of age of 26 who have joined their parents’ plan as a result of the law. But “3 million” is a problematic number.

I agree again that the "sub26er" number has been very, very difficult to nail down. Not only are there several different studies which used different methods to determine the figure, it's also an ever-moving target, and most of the studies are at least 2 years out of date by now anyway. Since it covers a very narrow age range, and pre-dates the exchanges (the "sub26er" rule went into effect back in 2010, I believe), this is a huge issue: Children who were as young as 15 when the provision went into effect have turned 19 by now, while young adults who were 22 - 25 have all aged out of the qualifying range.

In addition, even within the 19-25 range, there's been a lot of "churn"...what if you're 22 and on your parent's plan under the ACA provision, but then get a job with benefits through your employer? What if you have a falling out with your parents (or simply decide that you want to be more self-sufficient) and decide to purchase a QHP via the exchange directly? What if you get married to someone already covered and move off of your parents' plan?

For these reasons, I use a lower-end range of 1.6 million (1.4 million less than Sebelius), although I keep the 3.1 million figure at the high end since it's hard to tell. In this case, she's either matching or doubling what I have.

Sebelius: 17 million; Me: Between 16.3 - 20.2 million

’5 million in ACA-compliant off-market plans’

This is the most controversial part of Sebelius’s calculation. Off-market plans are sold directly by insurance brokers or insurance companies, meaning they do not qualify for Obamacare subsidies. But they do include the protections included in the law, such as guaranteed coverage for people with pre-existing conditions and a package of essential benefits.

There's actually two major questions here: First, how many off-exchange QHPs have been enrolled in so far, and second, should those off-exchange QHPs be counted at all, regardless of the number?

On the number itself, Sebelius is actually playing it very cautious, going with the CBO's early March estimate of around 5 million (almost identical to the 5 million people who had enrolled in QHPs on the exchanges by that point). However, that was before the late March/early April surge, when another 3 million exchange-based QHPs were enrolled in...and, as it happens, my own estimates, along with other studies by the RAND Corp. and so forth, strongly suggest that the off-exchange total has continued to match the exchange-based numbers fairly closely. This is why I have the off-exchange number down as roughly 8 million...3 million more than Sebelius stated.

Sebelius: 22 million; Me: Between 24.3 - 28.2 million

As an aside, I should note that Sebelius didn't even bother throwing in the small number of exchange-based SHOP enrollees or documented ACA-compliant off-exchange ESI's which I've documented; the two combine to only 156,000 people that I know of, so this is kind of a rounding error, but I'm including it anyway for the sake of completeness sake):

Sebelius: 22 million; Me: Between 24.5 - 28.4 million. (The graph shows the high end as 29 million due to rounding in some numbers above).

However, the bigger question Kessler poses isn't whether off-exchange QHPs are 5 million or 8 million, but whether they should be "counted" at all:

Given that Sebelius is touting “affordable coverage,” she should stick to citing the figures for the central parts of Obamacare — insurance bought on the exchanges and the expansion of Medicaid. 

In other words, Kessler is arguing that the off-exchange enrollments shouldn't "count" because of the "affordable coverage due to the ACA" specifications. Those 5-8 million off-exchange QHPs may be improvements in coverage (no recissions, no denials for pre-existing conditions, no gender-based discrimination, no annual/lifetime benefits cap, etc.), but Kessler is arguing they still don't qualify within the confines of Sebelius's statement.

In short, Kessler feels that the number Sebelius should have used was 8M exchange QHPs + 6M Medicaid + 1 or 2M Sub26ers, or about 15 million, leaving out the off-exchange enrollees entirely.

Here's the main area where I disagree with Kessler. In my view, every ACA-compliant healthcare policy issued to date is indeed more affordable due to the ACA, simply because there's no longer any chance of going into personal bankruptcy due to insane medical bills if you're covered by one. And while these 5-8 million enrollments may not have been done through the exchanges, they also wouldn't have included some of the ACA-required coverage areas if the law hadn't been passed.

HOWEVER, this is getting into a judgment call area, so I can't criticize Kessler too much over it. He ends up basically giving her a "Half True" score (2 Pinocchios out of 4).

And this is what the title of this entry is all about. "Fact Checking" is a very tricky business, because--shocking as it may seem--not all "facts" can be taken at face value; context has to be provided in most cases.

Going back to the first number at the top of this article (8 million on the exchange), if Kessler had asked Sebelius' spokesperson which specific 8 million enrollments she was referring to, they probably would have given the March/April HHS report as her source. If so, that wasn't true; only about 7.2 million of those enrollments have actually been paid. On the other hand, as I've noted, there have been an additional estimated 600K enrollments since 4/19, of which a good 500K+ have paid their first month's premiums by now...making the number around 7.6 - 7.7 million...which in turn means that Sebelius's "8 million" is actually pretty damned close.

So, does she get credit for telling the truth, or trashed for telling a lie?

If I make the statement "Tomorrow I will wash my car", but I break my arm and end up in the hospital for a week, does that mean that I lied? After all, I didn't end up washing my car, did I?

I don't agree with Kessler on all of his stories, but on this one, which is directly in my bailiwick, I give him kudos for threading the needle. I probably would have gone with 1 Pinocchio instead of 2, but that's just my take.

UPDATE: Thanks to Adam Goldstein for bringing this analysis of the numbers to my attention (as well as Glenn Kessler's) by the New England Journal of Medicine (which sure as hell has to be considered a Reliable Source). Their conclusion:

Taking all existing coverage expansions together, we estimate that 20 million Americans have gained coverage as of May 1 under the ACA (Figure 3FIGURE 3 Categories of Expanded Health Insurance Coverage under the Affordable Care Act (ACA).). We do not know yet exactly how many of these people were previously uninsured, but it seems certain that many were. Recent national surveys seem to confirm this presumption. The CBO projects that the law will decrease the number of uninsured people by 12 million this year and by 26 million by 2017. Early polling data from Gallup, RAND, and the Urban Institute indicate that the number of uninsured people may have already declined by 5 million to 9 million and that the proportion of U.S. adults lacking insurance has fallen from 18% in the third quarter of 2013 to 13.4% in May 2014.

However, these surveys may underestimate total gains, since some were fielded before the late March enrollment surge and do not include children. With continuing enrollment through individual marketplaces, Medicaid, and SHOP, the numbers of Americans gaining insurance for the first time — or insurance that is better in quality or more affordable than their previous policy — will total in the many tens of millions.

The NEJM's number, like Sebelius's, simplifies things a bit: 8 million exchange QHPs, 6 million Medicaid/CHIP, 1 million sub26ers, 5 million off-exchange QHPs = 20 million.

Their tally is 2 million less than Sebelius's due to the 1 million vs. 3 million discrepancy on the "sub26er" figure.

Their tally is another 3 million less than mine due to both the NEJM and Sebelius using the CBO's mid-March 5 million figure for off-exchange enrollments, even though there's strong evidence/data since that time to put this number closer to 8 million.

The main point here is that the NEJM is "siding" with both Sebelius and myself on the "should off-exchange QHPs count" question, stating:

Preoccupation with the individual marketplaces obscured another important effect of the ACA: increased enrollment outside the marketplaces. The law's new regulations affecting private health insurance that is sold to individuals and small employers in the United States protect consumers and small companies, whether they buy plans in the new ACA marketplaces or outside them in traditional insurance markets. This creates another entry point to coverage for people who previously would have faced exorbitant premiums or been shut out of the market altogether because of age or preexisting health conditions. And of course, the individual mandate creates added incentives for individuals to sign up. 

Exactly the point I was making the other day.