Philadelphia Inquirer misrepresents U of Penn ACA enrollment study (UPDATED x2)

I stumbled across an article in the Philadelphia Inquirer today which makes a rather odd claim in both the headline and the lede:

State sites outperform U.S. marketplace in Affordable Care Act signups

States running their own Affordable Care Act marketplaces enrolled more people in health insurance than those using the federal marketplace, according to an analysis by researchers at the University of Pennsylvania's Leonard Davis Institute.

Given the federal Website's dreadful October launch, that isn't exactly jaw-dropping news. But Penn's Health Insurance Exchange researchers were surprised to find that even after Healthcare.gov began working well in December, state-based marketplaces kept outperforming the federal site.

Huh. Interesting. They even specified that this was true "even after December".

Of course, there's a very good reason why they would be "surprised" at this finding...it's not even remotely close to being true.

Unfortunately, the Philly.com article doesn't provide a link to the actual U of Penn analysis, so I can't tell if the error is in the original research or if the reporter simply misinterpreted it. However, the story does make it clear that the cut-off point was the last official HHS report, which ran through April 19th:

With its money, Connecticut enrolled 79,192 people, spending about $50 per person enrolled. New Jersey signed up 161,775 for about $31 per person. Pennsylvania was the most economical, registering 318,077 enrollees at $22 per person.

Those numbers (79,192 for CT, 161,775 for NJ and 318,077 for PA) were the official exchange QHP enrollee counts as of 4/19, so we're not talking about the estimated 9K/day who've enrolled nationally since then.

So, let's look at the hard numbers this refers to by each monthly report:

• October: 79,391 State / 26,794 Fed / 106,185 Total
• 74.8% State Exchanges / 25.2% Federal Exchange

• November: 148,087 State / 110,410 Fed / 258,497 Total
• 57.3% State Exchanges / 42.7% Federal Exchange

• December: 729,513 State / 1,059,226 Fed / 1,788,739 Total
• 40.8% State Exchanges / 59.2% Federal Exchange

• January: 402,913 State / 743,158 Fed / 1,146,071 Total
• 35.2% State Exchanges / 64.8% Federal Exchange

• February: 261,335 State / 681,498 Fed / 942,833 Total
• 27.7% State Exchanges / 72.3% Federal Exchange

• March/April (partial): 952,346 State / 2,825,092 Federal / 3,777,438 Total
• 25.2% State Exchanges / 74.8% Federal Exchange

• Cumulative, 10/1/13 - 4/19/14: 2,573,585 State / 5,446,178 Federal / 8,019,763 Total
• 32.1% State Exchanges / 67.9% Federal Exchange
   

OK, as the article notes, in October, the 15 state-run exchanges did indeed kick the metaphorical butt of Healthcare.Gov due to HC.gov's legendary technical problems.

In November, with the worst of the problems at HC.gov resolved, things started to even out, but the state exchanges (representing 34% of the total population) were still vastly outperforming the federal exchange, which covered 36 states (representing 66%).

By January, however, the balance had shifted, and HC.gov enrolled almost 65% of the total--roughly its fair share, by total national population.

Finally, by the time the open enrollment period ended in March/April, the federal exchange was racking up 3 out of 4 total QHP enrollments.

So, the Philly.com story is quite simply flat-out wrong in both the headline claiming that the state-run exchanges "outperformed" the federal one (this was only true for the first 3 months) as well as in the opening paragraph claiming that they "enrolled more people" (this was true for 2 months but by December HC.gov was at over 50% of both monthly and total enrollments).

Heck, even if they meant in terms of cumulative enrollments for the entire October - April period, the article is still wrong; HC.gov ended up racking up 67.9% of total enrollees, slightly higher than what they would be "expected" to reach proportionately (66%).

Now, it's possible that the study is actually talking about the percent of the uninsured who enrolled in each state (which varies widely depending on population and demographics, of course). However, even using that as the basis, out of about 47.6 million people who were uninsured last fall, 14.8 million are served by the state-run exchanges...or 31.1%. I suppose if that's the basis for this claim, you could argue that HC.gov came up slightly short (67.9% of enrollments vs. 68.9% of uninsured population), but that would require a lot of goalpost moving and would still be a pretty thin argument.

The section quoted above makes it pretty clear that the study refers to exchange-based QHP enrollments only, so there shouldn't be any off-exchange or Medicaid data included here. I'd have to take a look at the study itself to be sure. Without that, I only have the article itself to go on, and it's very, very wrong.

Interestingly, not only is this the second time in the past 6 weeks that the Philly Inquirer has completely botched the numbers in an ACA story, both articles were written by the same reporter.

Neither story seems to have an agenda either in support of or opposition to the ACA, but they're both glaringly off in their primary claims, although the larger point made later in each article seems to be fairly on base.

UPDATE 8/24/14: Wow, that was fast!

I emailed Professor Baker (the U Penn prof referred to in the Philly.com article) and he wrote back almost instantly with a direct link to the actual research in question. There's 4 different studies listed, but 2 aren't relevant and one is outdated. The one I think that the Inquirer is referring to is the May study, Deciphering the Data: Final Enrollment Rates Show Federally Run Marketplaces Make Up Lost Ground at End of Open Enrollment.

This seems to be the key paragraph:

"Overall, more than 8.0 million people have enrolled and picked a plan through the exchanges, about 28% of all potential eligibles. We found that, on average, state-based marketplaces have had higher enrollment rates (32.5% of eligibles) than the federally facilitated ones (26.3%) or the partnership states (26.0%). The states retaining plan management functions within a federally facilitated marketplace have slightly lower rates than the other federally run ones (22.0% vs. 27.0%)."

Ah, ok. So, it's not the total population or the total uninsured population that we're going by here, but the total uninsured population eligible for QHPs.

If you use that as the basis, using the Kaiser Family Foundation's estimates from last October, after subtracting Undocumented Immigrants, those in the Medicaid Gap and those eligible for Medicaid, you get:

• State-Run Exchanges: 5,835,000 uninsured eligible for QHP enrollment
  • 2,573,585 / 5,835,000 = 44.1% enrolled
• Federal-Run Exchange: 16,623,000 uninsured eligible for QHP enrollment
  • 5,446,178 / 16,623,000 = 32.8% enrolled

OK, that kind-of, sort-of explains the "outperforms" reference in the headline, but it still in no way justifies the claim that the state exchanges "enrolled more people" than the federal exchange did.

Except...the numbers still don't make sense to me.

For one thing, technically anyone who’s a legal citizen and whose income is above a certain threshold is eligible to purchase a QHP via the exchanges even if they don’t qualify for a tax subsidy (I don’t see any reference to this in the study).

For another, those numbers are still vastly different from the 32.5% vs. 26% numbers claimed in the study.

Furthermore, even if we’re only talking about those eligible for tax subsidies, the numbers still don’t seem to add up. The next paragraph says:

 Within the federally run marketplaces, enrollment rates vary from 11% in South Dakota to 39% in Florida. 

South Dakota’s 13,104 enrollees supposedly represent 11% of those eligible in the state, but according to the Kaiser Family Foundation, SD had around 110K total uninsured, 69K eligible to purchase QHPs via HC.gov, and 40K eligible to receive a tax subsidy for doing so.

13,104 = 12% of the total uninsured, 19% of the QHP crowd and 33% of the tax subsidy crowd.

Assuming the KFF numbers are a bit different from the RWJF, it looks like SD is based on the total uninsured in the state.

However, in Florida, KFF says the numbers are 3.867 million total, 2.165 million eligible for QHPs and 1.273 million eligible for subsidies.

Florida's 983,774 = 25.4%, 45.4% and 77.3% respectively...none of which are close to the 39% claimed in the study.

I'm still missing something here. I've written Prof. Baker for further clarification and will update this with any further insight he's able to provide.

UPDATE x2 8/24/14: On the one hand, I give huge kudos to Professor Baker for his speedy responses. On the other hand...well, I think this summarizes everything that's wrong with journalism (and how we deal with it) these days:

Mr. Gaba--

I do my research.  I talk to reporters.  I write papers.  What the press does with my work is their business.  I don’t engage in debates.

--Tom Baker