Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Dec 28;29(1):128-136.
doi: 10.1093/jamia/ocab254.

Estimating the impact of engagement with digital health interventions on patient outcomes in randomized trials

Lyndsay A Nelson  1   2 , Andrew J Spieker  3 , Lindsay S Mayberry  1   2   4   5 , Candace McNaughton  6   7 , Robert A Greevy  3
Affiliations

Estimating the impact of engagement with digital health interventions on patient outcomes in randomized trials

Lyndsay A Nelson et al. J Am Med Inform Assoc. .

Abstract

Objective: Guidance is needed on studying engagement and treatment effects in digital health interventions, including levels required for benefit. We evaluated multiple analytic approaches for understanding the association between engagement and clinical outcomes.

Materials and methods: We defined engagement as intervention participants' response rate to interactive text messages, and considered moderation, standard regression, mediation, and a modified instrumental variable (IV) analysis to investigate the relationship between engagement and clinical outcomes. We applied each approach to two randomized controlled trials featuring text message content in the intervention: REACH (Rapid Encouragement/Education and Communications for Health), which targeted diabetes, and VERB (Vanderbilt Emergency Room Bundle), which targeted hypertension.

Results: In REACH, the treatment effect on hemoglobin A1c was estimated to be -0.73% (95% CI: [-1.29, -0.21]; P = 0.008), and in VERB, the treatment effect on systolic blood pressure was estimated to be -10.1 mmHg (95% CI: [-17.7, -2.8]; P = 0.007). Only the IV analyses suggested an effect of engagement on outcomes; the difference in treatment effects between engagers and non-engagers was -0.29% to -0.51% in the REACH study and -1.08 to -3.25 mmHg in the VERB study.

Discussion: Standard regression and mediation have less power than a modified IV analysis, but the IV approach requires specification of assumptions. This is the first review of the strengths and limitations of various approaches to evaluating the impact of engagement on outcomes.

Conclusions: Understanding the role of engagement in digital health interventions can help reveal when and how these interventions achieve desired outcomes.

Keywords: behavior intervention; digital technology; mobile health; randomized controlled trial; user engagement.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
Functionality for the interactive text messages delivered in the REACH and VERB interventions.
Figure 2.
Figure 2.
Illustration of three approaches to understand the role of engagement. In (A), engagement is treated as a predictor of the outcome. In (B), engagement is treated as a mediator between the intervention and the outcome. In (C), the effect of the intervention on the outcome varies between those who would engage with the intervention when assigned to it and those who would not (hence the term “hypothetical” engagement). The absence of an arrow between two variables presumes that no causal association is permitted.
Figure 3.
Figure 3.
Histograms depicting distribution of response rates in the REACH and VERB studies.
Figure 4.
Figure 4.
Point estimates and 95% confidence intervals for the never-engager causal effect and the engagement-compliant causal effect in adjusted models at lower and higher specifications for the sensitivity parameter, γ. The difference in treatment effects between the engagement compliant and the never engagers are derived from these estimates.
See this image and copyright information in PMC

References

    1. Lewis J, Ray P, Liaw ST. Recent worldwide developments in eHealth and mHealth to more effectively manage cancer and other chronic diseases - a systematic review. Yearb Med Inform 2016; 25 (01): 93–108. - PMC - PubMed
    1. Choi W, Wang S, Lee Y, et al. A systematic review of mobile health technologies to support self-management of concurrent diabetes and hypertension. J Am Med Inform Assoc 2020; 27 (6): 939–45. - PMC - PubMed
    1. Greenwood DA, Gee PM, Fatkin KJ, et al. A systematic review of reviews evaluating technology-enabled diabetes self-management education and support. J Diabetes Sci Technol 2017; 11 (5): 1015–27. - PMC - PubMed
    1. Marcolino MS, Oliveira JAQ, D'Agostino M, et al. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018; 6 (1): e23. - PMC - PubMed
    1. Hamine S, Gerth-Guyette E, Faulx D, et al. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res 2015; 17 (2): e52. - PMC - PubMed

Publication types