In an effort to improve quality and reduce costs, Medicare conducts “demonstration projects.”  These experiments in delivering medical care are tested in a few areas with the hope that, if they work, Medicare can apply them to the entire nation.

There’s just one problem.  As the Congressional Budget Office has noted, they seldom work.  After years of trying, Medicare has a hard time coming up with a demonstration project that improves the quality and cuts the cost of medical care.

Until now, that is.  As reported by amednews.com:

A Medicare test project that emphasized care coordination to keep patients from going back to the hospital scored lower readmission rates compared with similar regions without such a program in place.

A study on the Medicare quality improvement organization pilot in the Jan. 23/30 Journal of the American Medical Association reported that 30-day readmission rates in 14 communities were reduced by 5.7% over two years beginning in 2008. In an average locale serving 50,000 Medicare beneficiaries, the care coordination model would cost $1 million a year but also save $4 million by preventing return trips to hospitals, said lead author Jane Brock, MD, MSPH.

 Naturally, the folks who run Medicare are quite happy:

Following the success during the trial period, the Centers for Medicare & Medicaid Services has moved to expand these improvement groups nationwide by 2014. Researchers believe the program could save the system billions of dollars nationally.

It sounds almost too good to be true.  And if you look at the actual JAMA article, it probably is.

In this demonstration project, 14 communities participated in a system to improve “care transitions”—i.e. moving from the hospital to one’s home or nursing home—for Medicare patients.  The researchers found that in the communities in the demonstration project hospital readmissions were reduced .56 per 1000 per quarter versus communities not in the demonstration project.

However, to test whether that estimate of .56 is accurate, one uses a “confidence interval.”  If the confidence interval is pretty close to the estimate—let’s say, in this case, it had been .50 on the low end and .60 on the high end—then researchers could be pretty confident that the estimate is accurate.

Here’s the rub:  The actual confidence interval is 0.05 to 1.07.  That’s a pretty wide range around .56.  Worse, 0.05 on the low end is very close to zero, meaning that it’s possible that care transitions may have almost no effect on hospital readmissions.

It’s certainly not a good enough confidence interval for researchers to be saying that the project could save billions nationwide.  A more accurate statement would be it could save billions but it might also save next to nothing.

Of course, that kind of statement doesn’t make the folks at CMS too happy.

For more on why Medicare demonstration projects don’t work, see this excellent piece by Megan McArdle.