…and Experience Comes from Bad Judgment
In Tuesday's post, I gave full rein to my fears about the possible faults and flaws with current federal Medicare quality improvement demonstrations. While I stand by what I wrote, I felt the immediate guilt of being caught stones in hand in a pretty drafty glass house.
Over the last 15 years I have participated in or directed (from the funder side) more projects than I can recall, efforts to demonstrate and disseminate improved models of care. And while I will stack the Foundation's record up to anyone's, my contention that most things don't work unfortunately also applies to us. Most of the demonstration efforts we have supported have failed to influence practice widely, therefore failing to meet OUR objective for them: to improve the health of older Americans.
During those 15 years we have learned more and more about what is needed to drive change in health care delivery, one painful lesson at a time. We have learned that good ideas are not enough. Even those ideas that have gone through the development and testing phase to successfully answer the question Does it work? still need to publicize their existence and set up technical assistance and tools for their adoption. They need conducive attitudinal, regulatory, and financial conditions under which they can grow and flourish. "Experiences" abound at every level. So, in the spirit of sharing lessons learned, here is a non-exhaustive list:
Be focused. When I joined the Foundation, we were just winding up evidence collection for our Generalist Physician Initiative. It was an incredibly foresighted series of demonstrations that placed additional nursing and social work staff into primary care practices, either co-located or virtually, to help meet the complex needs of older adults. Some good models were in the group and some good ideas, but I can't say we learned much from the initiative as a whole. We had no clarity about whether we were prototyping models, evaluating models for cost and quality impact, or just stirring the pot. The book that resulted is still available on Amazon. But if there is anything that makes me worry about the Innovation Challenge grant program, it was this experience. We had the privilege of supporting some of the smartest and best people I know, and yet somehow we made the collective LESS than the sum of the parts.
Overdesign for success. No matter how simple or cheap your model, as soon as you prove that it works, even if it reduces costs, people will begin asking if they can drop parts of the model to make it cheaper. On the other hand, if your model doesn't work, there will be no questions at all. When we designed the IMPACT model, we knew that we were "putting the kitchen sink" into the intervention, but we didn't want to contribute to therapeutic nihilism about depression treatment by producing yet another failure. It is better to have an argument about how to implement more efficiently than to have no argument.
Plan for challenges. Innovations are fragile and need every advantage they can get. Two rules: 1) Buy stand-by capacity to deliver the intervention; i.e., make sure you have personnel whose only job is to do the new job. I've long regretted that I didn't allocate $300,000 more to the home hospital multi-site trial. With the extra $100,000 per site, there would have been clinicians on stand-by when referrals came for "home hospital care" for common geriatric conditions. Instead, home care clinicians were often unavailable because they had gotten involved in other work. Without the stand-by capacity, we had a vicious cycle where referrals were not picked up, referral rates dropped off, and it became harder and harder to keep clinical capacity available to care for people in their homes. 2) Be sure that you have contingency plans to recruit more participants when it turns out (as it always does) that they are scarcer than they seemed in the planning phase or less willing to participate than you had hoped. Never, never, never trust anyone's projected numbers.
Remember your audience. The objective of demonstration and dissemination is to win the hearts and minds of some audience of stakeholders. It's not to prove that you are right (or smart). When we started working with the NCOA around evidence-based models of health promotion, I felt that the task was to take published scientifically adequate evidence and translate it into "real world" programs that could be hosted effectively by community organizations. Given that the programs’ elements had already been tested, I thought more testing was superfluous, and I didn't want the Foundation to pay for it (again). I was wrong. It wasn't until the Administration on Aging sponsored evaluations of what we developed, Healthy Ideas, Healthy Moves, and Healthy Eating, that they really took off.
Design with the end in mind. Unless all you want is a publication, it's a bit late to wait to see if your model works to start developing training materials, collecting patient cases and stories, building awareness among stakeholders, and all the other tasks essential for replication and spread. You have to work ahead of the evidence. A risk, yes, but necessary.
Buy the best evaluation you can afford. While the cynic in me likes to joke that people don't care (much) about evidence, many really do. You do yourself. A good evaluation is expensive, very expensive, but it is an investment that you can draw upon for years and years. A true experiment with people randomized to your new intervention or some control condition is the gold standard for demonstrating a causal (rather than coincidental) relationship between intervention and outcome. Adequate sample size gives you your best chance to find a true effect and reject illusions. I have never gone wrong with a good evaluation. I have often gone wrong wanting a cheap alternative. If you don't believe in what you are doing enough to want to do it right, are you sure you should do it at all?
Remember the innovator's dilemma and sometimes forget your audience. The idea of the innovator's dilemma as developed by business scholar Clayton Christenson is that staying close to an audience (customers) can actually keep you from innovating in directions that may ultimately be very useful because they are not currently valued. Plenty of health care stakeholders still don't want to reduce readmissions (that's REVENUE!) much less reduce admissions or improve primary care (money loser!). Yet those things need to be done. As a mission-driven funder of innovators, we need to keep an eye on what is feasible and attractive today, but also what is good for people and society that perhaps we can make work in some future reality.
Our objective is not just to live in and maybe move some money around this world. We want to change it for the better.