Section 9: Conduct of Pragmatic Research

9.1 Introduction

Congratulations on starting a new pragmatic research project! This module is designed to help with practical steps in launching and conducting your study. A first step is to ensure you and your team have oriented to the context and setting in which your research will be conducted. A next step is to identify and prepare any technology and infrastructure needed.

CONTENT NEEDED:  different ways of implementing pragmatic trials (through the EHR, in real-world practice using hybrid designs, etc), protocol refinement,  adaptation/flexibility (PRECIS-2 reference flexibility in adherence domain); fidelity to protocol and protocol refinement; making use of real-world data and using for analysis; engaging stakeholders and partners in the conduct of research; process evaluation.

References

  • Kilbourne AM, Braganza MZ, Bowersox NW, et al. Research Lifecycle to Increase the Substantial Real-world Impact of Research: Accelerating Innovations to Application. Med Care. 2019;57 Suppl 10 Suppl 3(10 Suppl 3):S206-S212. doi:10.1097/MLR.0000000000001146
  • Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality Enhancement Research Initiative Implementation Roadmap: Toward Sustainability of Evidence-based Practices in a Learning Health System. Med Care. 2019;57 Suppl 10 Suppl 3(10 Suppl 3):S286-S293. doi:10.1097/MLR.0000000000001144

9.2 Community Engagement

[CONTENT NEEDED – what is needed here, separately from section 6?]

For additional resources about engaging community members and other partners, see Section 6: Engaging Potential Adopters, Influencers and End Users.

9.3 Implementation Strategies

[CONTENT NEEDED – summary below from Mittman 2021 keynote handout, need permission if we want to use this.]

Implementation and implementation research are integral to pragmatic research and the ability of pragmatic trials to achieve their goals. Pragmatic research and pragmatic researchers must account for implementation phenomena and implementation goals in study design, conduct and follow-up. The need to attend to implementation ranges from the dependence of intervention effectiveness on implementation patterns and processes (and hence the need to explicitly consider implementation in formulating research questions and study designs) to the critical role of implementation in facilitating intervention use and, therefore, achievement of intervention and pragmatic research goals. Frameworks and approaches from diverse fields can inform appropriate consideration of implementation in pragmatic research, including fields such as improvement science, learning health systems, systems thinking and the study of complex health interventions.

References

  • Fernandez ME, Ten Hoor GA, van Lieshout S, et al. Implementation Mapping: Using Intervention Mapping to Develop Implementation Strategies. Front Public Health. 2019;7:158. Published 2019 Jun 18. doi:10.3389/fpubh.2019.00158

  • Powell BJ, Beidas RS, Lewis CC, et al. Methods to Improve the Selection and Tailoring of Implementation Strategies. J Behav Health Serv Res. 2017;44(2):177-194. doi:10.1007/s11414-015-9475-6

  • Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139. Published 2013 Dec 1. doi:10.1186/1748-5908-8-139
  • Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice. Implement Sci. 2017;12(1):125. Published 2017 Nov 3. doi:10.1186/s13012-017-0657-x
  • Powell BJ, Fernandez ME, Williams NJ, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front Public Health. 2019;7:3. Published 2019 Jan 22. doi:10.3389/fpubh.2019.00003
  • Ritchie MJ, Parker LE, Edlund CN, Kirchner JE. Using implementation facilitation to foster clinical practice quality and adherence to evidence in challenged settings: a qualitative study. BMC Health Serv Res. 2017;17(1):294. Published 2017 Apr 20. doi:10.1186/s12913-017-2217-0
  • Ritchie MJ, Parker LE, Kirchner JE. From novice to expert: a qualitative study of implementation facilitation skills. Implement Sci Commun. 2020;1:25. Published 2020 Feb 25. doi:10.1186/s43058-020-00006-8
  • Guise JM, Savitz LA, Friedman CP. Mind the Gap: Putting Evidence into Practice in the Era of Learning Health Systems. J Gen Intern Med. 2018;33(12):2237-2239. doi:10.1007/s11606-018-4633-1
  • Weinfurt KP, Hernandez AF, Coronado GD, et al. Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol. 2017;17(1):144. Published 2017 Sep 18. doi:10.1186/s12874-017-0420-7

Videos

Implementation and Conduct of Pragmatic Trials: The Intersection of Quality Improvement, Learning Health Systems, and System Level Change

Implementation Mapping: A Promising and Innovative Method to Design and Select Implementation Strategies for Firearm Safety Promotion in Pediatric Primary Care

Planning for Practice Facilitation in Your Research Proposal: What You Need to Know

9.4 Ethics in Conduct of Pragmatic Research

[General ethics content?]

CONTENT from DREXEL 2021 – need permission if we want to use this:

Pragmatic trials often aim to take advantage of technology for outreach and implementation, with the goals of broadening outreach and reducing burden on participants. One benefit of such approaches is the reduction in barriers presented by in-person consent processes. Different methods of outreach and consent, however, result in different sub-populations of participants who enroll. Both of these features can influence the estimates of treatment effectiveness that may result, as well as generating ethical concerns.

References

  • Sugarman J, Califf RM. Ethics and regulatory complexities for pragmatic clinical trials. JAMA. 2014;311(23):2381-2382. doi:10.1001/jama.2014.4164

Videos

Implementing Pragmatic Trials via Electronic Platforms: Practical and Ethical Considerations for Consent, Participation, and Analysis

9.5 Fidelity and Adaptation in Pragmatic Research

[INTRO NEEDED]

Adaptation and complexity

[NOTE: this content from Moore 2021, need permission to use] Complex systems approaches to intervention science are increasingly advocated. However, despite a growing abundance of conceptual publications, there have to date been few attempts to consider in practical terms what a complex systems lens means for intervention researchers. This talk discusses the influence of complex systems thinking within historical and ongoing guidance development for intervention researchers. It will argue that whole systems evaluations may be neither attainable, nor necessary and that acknowledgment of complexity does not mean that real-world evaluations must investigate all facets of complexity. However, a systems lens may add value to intervention science through framing issues such as how fidelity is conceived, aspects of context which matter for intervention effects to transfer across contexts, and how intervention effects may build or diminish over time in real world settings. The talk will introduce new MRC-NIHR funded ADAPT guidance, which draws upon complex systems perspectives to provide guidance in thinking through and undertaking the adaptation of interventions for new contexts

Cultural Adaptations

[NOTE: THIS CONTENT FROM PHIMPASONE-BRADY, 2021 – NEED PERMISSION TO USE] The aim of cultural adaptation is to protect the scientific integrity of evidence-based treatments (EBTs) and promote dissemination by promoting the external and ecological validity of studies.The ultimate aim is to reduce health disparities by making EBTs broadly available to diverse cultural groups.

Cultural adaptation integrates multicultural contexts and EBTs to consider culture and context systematically (historical, economic, ecological, and political influence). Several frameworks exist to support cultural adaptation:

  • Cultural Sensitivity Framework (Resnicow et al. 1999)
  • Cultural Adaptation Process Model (Domenech Rodríguez et al. 2004)
  • Cultural Adaptation Framework (Barrera et al. 2013)

References

  • Escoffery C, Lebow-Skelley E, Haardoerfer R, et al. A systematic review of adaptations of evidence-based public health interventions globally. Implement Sci. 2018;13(1):125. Published 2018 Sep 26. doi:10.1186/s13012-018-0815-9
  • Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. Published 2013 Oct 2. doi:10.1186/1748-5908-8-117
  • Chambers DA, Norton WE. The Adaptome: Advancing the Science of Intervention Adaptation. Am J Prev Med. 2016;51(4 Suppl 2):S124-S131. doi:10.1016/j.amepre.2016.05.011
  • Kirk MA, Moore JE, Wiltsey Stirman S, Birken SA. Towards a comprehensive model for understanding adaptations’ impact: the model for adaptation design and impact (MADI). Implement Sci. 2020;15(1):56. Published 2020 Jul 20. doi:10.1186/s13012-020-01021-y
  • Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci. 2021;16(1):36. Published 2021 Apr 7. doi:10.1186/s13012-021-01105-3
  • Roscoe JN, Shapiro VB, Whitaker K, Kim BKE. Classifying Changes to Preventive Interventions: Applying Adaptation Taxonomies. J Prim Prev. 2019;40(1):89-109. doi:10.1007/s10935-018-00531-2
  • Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58. Published 2019 Jun 6. doi:10.1186/s13012-019-0898-y
  • Moore GF, Audrey S, Barker M, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. Published 2015 Mar 19. doi:10.1136/bmj.h1258
  • Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3-4):267-276. doi:10.1007/s10464-009-9229-9
  • Moore GF, Evans RE, Hawkins J, et al. From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation. Evaluation (Lond). 2019;25(1):23-45. doi:10.1177/1356389018803219
  • Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017;3:132-135. doi:10.1016/j.ssmph.2016.12.005
  • Moore G, Campbell M, Copeland L, et al. Adapting interventions to new contexts-the ADAPT guidance. BMJ. 2021;374:n1679. Published 2021 Aug 3. doi:10.1136/bmj.n1679
  • Mazzucca S, Arredondo EM, Hoelscher DM, et al. Expanding Implementation Research to Prevent Chronic Diseases in Community Settings. Annu Rev Public Health. 2021;42:135-158. doi:10.1146/annurev-publhealth-090419-102547
  • Resnicow K, Baranowski T, Ahluwalia JS, Braithwaite RL. Cultural sensitivity in public health: defined and demystified. Ethn Dis. 1999;9(1):10-21. https://www.jstor.org/stable/45410142
  • Domenech-Rodríguez M, Wieling E. Developing culturally appropriate, evidence-based treatments for interventions with ethnic minority populations. In: Rastogi M, Wieling E, eds. Voices of Color: First-Person Accounts of Ethnic Minority Therapists. SAGE Publications, Inc. ; 2005:313-333. doi:10.4135/9781452231662.n18
  • Barrera M Jr, Castro FG, Strycker LA, Toobert DJ. Cultural adaptations of behavioral health interventions: a progress report. J Consult Clin Psychol. 2013;81(2):196-205. doi:10.1037/a0027085
  • Barrera M Jr, Berkel C, Castro FG. Directions for the Advancement of Culturally Adapted Preventive Interventions: Local Adaptations, Engagement, and Sustainability. Prev Sci. 2017;18(6):640-648. doi:10.1007/s11121-016-0705-9
  • Baumann A, Domenech Rodríguez M, Parra-Cardona JR. Community-based applied research with Latino immigrant families: informing practice and research according to ethical and social justice principles. Fam Process. 2011;50(2):132-148. doi:10.1111/j.1545-5300.2011.01351.x
  • Bernal G, Domenech MM, eds. Cultural Adaptations: Tools for Evidence-Based Practice with Diverse Populations. 1st ed. American Psychological Association; 2012. doi:10.1037/13752-000
  • Cabassa LJ, Baumann AA. A two-way street: bridging implementation science and cultural adaptations of mental health treatments. Implement Sci. 2013;8:90. Published 2013 Aug 19. doi:10.1186/1748-5908-8-90
  • Domenech Rodríguez MM, Baumann AA, Schwartz AL. Cultural adaptation of an evidence based intervention: from theory to practice in a Latino/a community context. Am J Community Psychol. 2011;47(1-2):170-186. doi:10.1007/s10464-010-9371-4

Videos

Using Frame and MADI Frameworks to Guide and Track Adaptations

Reconceptualizing Sustainability and Adaptation: From Static to Dynamic

Implementation and Conduct of Pragmatic Trials: The Intersection of Quality Improvement, Learning Health Systems, and System Level Change

Multi and Mixed Methods Approaches for Documenting and Analyzing Adaptations in Real-World Studies

Map2Adapt: A Roadmap to Plan for Adaptations

Making Complexity Pragmatic Again: Practical Steps to Systems Mapping and Modeling

Cultural Adaptations of Evidence-Based Interventions to Fit to Context

Understanding and Adapting to Complexity in Real-World Contexts

Process Evaluation and Adaptations in Complex Trials

9.6 Data Collection and Management

Note: there is also content about analysis in section 5 (study design), in the section on analysis methods.

[NOTE: THIS CONTENT FROM VOCK 2021, NEED PERMISSION TO USE] Existing/secondary data present an important resource which can be used at many points in the life cycle of pragmatic research, including for planning and trial design, participant recruitment, endpoint ascertainment, calibration of treatment effects, among others. However, existing/secondary data should be interrogated for not only what it includes but also what it systematically does not capture. When possible, the limitations of existing/secondary data should be ameliorated in the design and analysis plan. Many perceived weaknesses of existing and secondary data such as patient heterogeneity, measurement error of covariates, etc. should be reframed as strengths for pragmatic research.

References

  • Kahn MG, Callahan TJ, Barnard J, et al. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. EGEMS (Wash DC). 2016;4(1):1244. Published 2016 Sep 11. doi:10.13063/2327-9214.1244

  • Klann JG, Joss MAH, Embree K, Murphy SN. Data model harmonization for the All Of Us Research Program: Transforming i2b2 data into the OMOP common data model. PLoS One. 2019;14(2):e0212463. Published 2019 Feb 19. doi:10.1371/journal.pone.0212463

  • Prats-Uribe A, Sena AG, Lai LYH, et al. Use of repurposed and adjuvant drugs in hospital patients with covid-19: multinational network cohort study [published correction appears in BMJ. 2021 May 21;373:n1277]. BMJ. 2021;373:n1038. Published 2021 May 11. doi:10.1136/bmj.n1038
  • Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. 2017;106(1):1-9. doi:10.1007/s00392-016-1025-6
  • Sidebottom AC, Sillah A, Vock DM, et al. Assessing the impact of the heart of New Ulm Project on cardiovascular disease risk factors: A population-based program to reduce cardiovascular disease. Prev Med. 2018;112:216-221. doi:10.1016/j.ypmed.2018.04.016
  • Sidebottom AC, Sillah A, Miedema MD, et al. Changes in cardiovascular risk factors after 5 years of implementation of a population-based program to reduce cardiovascular disease: The Heart of New Ulm Project. Am Heart J. 2016;175:66-76. doi:10.1016/j.ahj.2016.02.006
  • Ru B, Yao L. A literature review of social media-based data mining for health outcomes research. In: Bian J, Guo Y, He Z, Hu X, editors. Social web and Health Research: benefits, limitations, and best practices. Cham: Springer International Publishing; 2019. pp. 1–14.
  • Taylor J, Pagliari C. Mining social media data: How are research sponsors and researchers addressing the ethical challenges? Research Ethics. 2017;14(2):1-39. doi:10.1177/1747016117738559
  • Reno JE, Ong TC, Voong C, et al. Engaging Patients and Other Stakeholders in “Designing for Dissemination” of Record Linkage Methods and Tools. Appl Clin Inform. 2023;14(4):670-683. doi:10.1055/a-2105-6505

Videos

The Benefits and Challenges of Leveraging Existing and Secondary Data for Pragmatic Research: A Case Study of Evaluating the Effect of Living Kidney Donation on Long-Term Outcomes

Planning Your Data Collection Strategies: Electronic Health Data

Building the Tower of Babel – Tricks and Traps in Harmonizing EHR Data

Using Population-Based Data in Secondary Analysis

Data Quality Assessment Issues and Methods for Secondary Data Use

Analyzing Real World Data: Methods for Linking Records Across Disparate Data Sources

Opportunities for Using Healthcare Claims Data for Pragmatic Sustainability Assessments

Digital Health Data Access, Management, and Use

Mining and Analyzing Data from Social Media Data Sources

9.7 Patient-Reported Outcomes

[NEED CONTENT FOR THIS SECTION]

References

  • Weldring T, Smith SM. Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs). Health Serv Insights. 2013;6:61-68. Published 2013 Aug 4. doi:10.4137/HSI.S11093
  • Carle AC, Cella D, Cai L, et al. Advancing PROMIS’s methodology: results of the Third Patient-Reported Outcomes Measurement Information System (PROMIS(®)) Psychometric Summit. Expert Rev Pharmacoecon Outcomes Res. 2011;11(6):677-684. doi:10.1586/erp.11.74

Videos

Patient Reported Measures: On the Ground Collection, Implementation and Clinical Workflows

9.8 Analysis and Reporting

 

[NEED CONTENT]

References

  • Rudd BN, Davis M, Beidas RS. Integrating implementation science in clinical research to maximize public health impact: a call for the reporting and alignment of implementation strategy use with implementation outcomes in clinical research. Implement Sci. 2020;15(1):103. Published 2020 Nov 25. doi:10.1186/s13012-020-01060-5
  • Richesson RL, Green BB, Laws R, et al. Pragmatic (trial) informatics: a perspective from the NIH Health Care Systems Research Collaboratory. J Am Med Inform Assoc. 2017;24(5):996-1001. doi:10.1093/jamia/ocx016

Videos

Methods for Reporting and Aligning Implementation Strategies with Implementation Outcomes