A May 2, 2013 article from the New England Journal of Medicine (Baicker, K., et al., The Oregon Experiment – Effects of Medicaid on Clinical Outcomes) is receiving a great deal of attention because it is being interpreted as an early indicator of how the Affordable Care Act’s expanded Medicaid coverage is likely to affect health outcomes for currently uninsured Americans. If you believe the coverage, the study found that providing Medicaid to the uninsured resulted in only marginal benefits which are insufficient to justify the ACA’s additional cost. (See here, here, here, and here, for just a handful of the articles reporting on the study.)Examined closely, however, the study isn't strong enough to support policy decisions. Here's why...
Unfortunately, most observers seem to be missing the all-important fact that the study does not provide a clean comparison of insured and uninsured populations. Although it seems possible that the underlying data could be analyzed to address that vital question, the current article doesn’t and anyone who says it does is either advancing an agenda or simply not reading closely enough.
For background, in 2008, the state of Oregon employed a lottery system to offer Medicaid coverage to approximately 30,000 members of a 90,000 person waiting list. This created a unique opportunity to compare the effects of Medicaid enrollment featuring randomized assignment to investigative (i.e. lottery winners) and control (i.e. lottery losers) arms.
To really understand why the study doesn’t quite merit the attention it’s receiving, you have to dig pretty deeply into the separately downloaded Supplemental Appendix. Specifically, Table S9 on page 34 of the Appendix provides details on insurance coverage for the control and investigative arms. It shows why this data, at least as it’s presented in the current article, should not serve as the basis for any major policy decisions.
Media coverage would lead one to believe that everyone who won the lottery was on Medicaid throughout the study and that none of those who lost were. In fact, Table S9 shows that over the course of the study, only 42.6% of people who won the lottery were ever on Medicaid; by comparison, 18.5% of those who lost the lottery were on Medicaid at some point.
Even this figure overstates the differences between the two groups because; in a phenomenon known as churn, an individual’s (or family’s) insurance status changes over time. For context, separate research has shown that the dropout rate for adults enrolled in Medicaid is 20% at six months, 43% at 12 months, and 55% at 23 months. (Saunders, M., et al., Journal of General Internal Medicine. Jan 2009).
The effects of churn are evident in the details of Table S9 as well. Specifically, average duration of Medicaid enrollment for all lottery winners was only 6.76 months out of a study period of 25 months while average duration for lottery losers was 2.6 months. Thus, lottery winners were, on average, enrolled just a little over four months longer than the control group.
In addition to churning into or out of Medicaid, people also gain or lose health insurance in the private market. At the time the interviews for this study were conducted, 14.7% of the control group and 14.3% of the lottery-winning group had private insurance. The presence of private insurance would have prevented Medicaid enrollment under any circumstance, so this shows that a considerable minority of people who were eligible for Medicaid at the outset of the study had found private coverage by the time they were interviewed.
One of the most straightforward measurements in the study is that, at the time of interview, the percentage of people in the control group with any form of health insurance was 35.8% as opposed to 46.9% among lottery winners. Unfortunately, the study does not provide information about the duration of “any insurance” for either group but that, relatively small, 11% incremental difference at a single point in time is arguably the best method for comparison between the two groups.
Summing up: the majority of lottery winners did not receive Medicaid while many in the control group did and the average duration of enrollment was relatively short for both groups. At the end of the study, there was an essentially equivalent chance of having private insurance and members of the control group were 76% as likely to have some form of coverage as were the lottery winners.
There are some other questionable analytical choices: for example, in a population where 72% of participants were under 50 years old, why use hypertension, diabetes, high cholesterol, and depression as primary measurements when, with the exception of depression, these are predominately conditions of late middle age? The problem here is one of statistical power: to take a more extreme example, if researchers were trying to measure the impact of health insurance on children, it wouldn’t make much sense for them to use congestive heart failure as a measurement. Although pediatric patients experience CHF, there are many other conditions that occur at a higher rate, making cross-group comparisons easier. With this in mind, it may be not be surprising that depression, which is the “youngest” of the four studied conditions (according to the National Institutes of Health, the average age of onset is 32), also showed the clearest benefit for Medicaid enrollment.
It’s possible that the underlying data could still serve the purpose that everyone wants it to – i.e. provide a clear-cut analysis on the benefits of having or not having Medicaid coverage. As presented in this article, however, the data is far too muddy to prove or disprove much of anything.