How Can We Know if a Digital Health Intervention Works? Reconsidering Success Metrics and Timelines

By Amy Bucher, PhD is Behavior Change Design Director at Mad*Pow. She tweets at @amybphd.  She is a member of the Society of Behavioral Medicine.

We talk about the potential of digital interventions to revolutionize healthcare. They’re scalable, portable, personal, and pack consumer appeal. But we still aren’t able to say with much certainty under what conditions they work, if any, particularly for some of the populations who could benefit most: The chronically ill, people with significant transportation, cost, or other barriers to traditional healthcare access, caregivers who don’t have formal medical training but want to monitor and assist their loved ones. If we could just figure out what interventions—or even what types of interventions—are clinically efficacious for outcomes that matter, we could recommend the right tools for each person’s interests and needs. And we may even be able to get health plans to pay for them.

So why don’t we know which digital health intervention works? One issue is that we need to figure out how to measure behavior change outcomes reliably. Another major problem is that the digital health industry is hyper-focused on short-term metrics at the expense of more meaningful longer-term ones. Companies look at metrics like sign ups, logins, and clicks to determine whether a product is successful; longer-tail outcomes, like health changes or cost reductions, often follow well after a product has been deemed a hit or failure.

This short-term focus is understandable. It’s a remnant of an advertising-driven internet, where clicks translated directly to revenue. Typical business models urge a focus on short term successes, with quarterly and annual reporting used as the basis for a company’s valuation and appeal, and venture capital funding raised in rounds intended to sustain a business for a few years at most. It may feel risky to entrepreneurs to take a longer view of success, especially in a market where technology can evolve quickly and dramatically.

Fortunately, the most promising future revenue models in digital health include players like pharmaceutical companies, hospitals, health systems, or possibly even government via CMS or the VA. These organizations have a vested interest in making sure solutions are truly efficacious, and they’re not likely to pay for ones that aren’t. To determine which ones those are, we must shift how we think about determining whether digital health interventions work.

It’s time to think about measuring the outcomes that really matter. These may take weeks, months, or even years to come about (hence the term lagging indicator often used to describe them). These outcomes may include:

• Biometric changes: Reductions in blood pressure, cholesterol, A1C, weight, and other biometric measures, which often require consistent behavior change over time.
• Productivity improvements: Employer customers of digital health solutions see value when employees are being better able to fulfill their professional roles because they’re not sidelined with health problems
• Healthcare cost reductions: If digital health makes people healthier, they should eventually spend less money annually on healthcare. Data suggests that more activated patients generate less health care cost, and that this may be particularly true for hospitalizations and emergency care. However, improved ownership of health may also drive cost increases in the short term associated with preventative care and medication adherence, and higher lifetime costs due to increased longevity). We therefore urge looking at healthcare costs on an annual basis rather than a lifetime basis, and considering that some healthcare costs such as pharmacy spend may enable lower spend elsewhere in the system.

Fortunately, we can do more than just wait patiently hoping to see positive results. Here’s how we can plan for success:

• Create an outcomes map as part of digital health intervention development. An evidence-based intervention relies on a logic model that can be made explicit. Begin by outlining the specific long-term metrics of importance, then trace backwards through the events that must happen chronologically for those outcomes to occur. Then, detail how you’ll measure that each step is taking place. A chronological outcomes map will give you an opportunity to monitor your intervention’s progress and iterate on the intervention if the data suggests necessary prerequisites to outcomes are not occurring or not occurring at a sufficient rate.

• Leverage population data and statistical models to boost confidence. If you have access to existing data on a digital health intervention, you may be able to statistically predict lagging indicators like biometric changes based on leading indicators like intervention usage—or, uncover that some leading indicators are poor predictors of outcomes and should therefore be dropped from consideration. Similarly, if you have access to health plan or other population data, you may be able to examine claims data alongside known patterns of intervention usage and determine whether there is any relationship between the two. Statistical investigations such as these offer another method of using leading indicators to gain confidence about the intervention’s likelihood of success.
• Pair quantitative data with qualitative insights. Both health science and corporate business have a bias toward numbers, “hard data.” But qualitative insights gained from talking to digital health users can add enormous value to your understanding of an intervention’s efficacy. Do people like using it? Do they intend to continue using it? What’s changed in their behaviors or attitudes as a result of using it? People’s answers to questions like these can offer valuable insights into the results they might see as well as potential barriers to success that may be addressable in a product iteration.

Ultimately, the stakeholders in the system who purchase digital health products want to see a return on investment. As more companies enter the digital health space, the ability to demonstrate meaningful long-term outcomes will distinguish the survivors from the forgotten.