Section 7: Pragmatic Outcomes, Measures, & Data Collection

7.1 Selecting Pragmatic Research Outcomes and Measures

In pragmatic research, to the extent possible, the outcomes and measures selected should align with the conceptual, theoretical, and/or process framework(s) guiding the research AND the needs, perspectives, and relevant metrics for success held by patients and other stakeholders.

The timing, frequency, and comprehensiveness of data collection is also important, and should be planned with respect to the study design, burden to respondents, research team resources, and consideration of plans to test for mediation, moderation and generalization/heterogeneity of effects.

References

  • Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45(2):237-243. doi:10.1016/j.amepre.2013.03.010
  • Weiner BJ, Lewis CC, Stanick C, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. Published 2017 Aug 29. doi:10.1186/s13012-017-0635-3
  • Stanick CF, Halko HM, Nolen EA, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2021;11(1):11-20. doi:10.1093/tbm/ibz164
  • Rabin BA, Purcell P, Naveed S, et al. Advancing the application, quality and harmonization of implementation science measures. Implement Sci. 2012;7:119. Published 2012 Dec 11. doi:10.1186/1748-5908-7-119
  • Moullin JC, Sklar M, Green A, et al. Advancing the pragmatic measurement of sustainment: a narrative review of measures. Implement Sci Commun. 2020;1:76. Published 2020 Sep 17. doi:10.1186/s43058-020-00068-8
  • Moullin JC, Sklar M, Ehrhart MG, Green A, Aarons GA. Provider REport of Sustainment Scale (PRESS): development and validation of a brief measure of inner context sustainment. Implement Sci. 2021;16(1):86. Published 2021 Aug 30. doi:10.1186/s13012-021-01152-w

Videos

Pragmatic Measures and Methods: Approaches Based on the PRISM and RE-AIM Framework

Pragmatic Measurement of Sustainment

7.2 Bias and Health Equity in Pragmatic Research Outcomes

Text needed.

References

  • Lorenc T, Petticrew M, Welch V, Tugwell P. What types of interventions generate inequalities? Evidence from systematic reviews. J Epidemiol Community Health. 2013;67(2):190-193. doi:10.1136/jech-2012-201257
  • Sadare O, Williams M, Simon L. Implementation of the Health Equity Impact Assessment (HEIA) tool in a local public health setting: challenges, facilitators, and impacts. Can J Public Health. 2020;111(2):212-219. doi:10.17269/s41997-019-00269-2
  • Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100 Suppl 1(Suppl 1):S40-S46. doi:10.2105/AJPH.2009.184036
  • Shelton RC, Adsul P, Oh A. Recommendations for Addressing Structural Racism in Implementation Science: A Call to the Field. Ethn Dis. 2021;31(Suppl 1):357-364. Published 2021 May 20. doi:10.18865/ed.31.S1.357
  • Shelton RC, Chambers DA, Glasgow RE. An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front Public Health. 2020;8:134. Published 2020 May 12. doi:10.3389/fpubh.2020.00134

Videos

Assessing and Enhancing Reach and Representativeness

Advancing Research on Sustainability and Health Equity in Implementation Science

7.3 Pragmatic Research Outcomes – Key Questions

To help answer these questions, you may want to refer and complete the exercises on dissemination-implementation.org.

7.3.1

What outcomes matter most to our stakeholders? What information do they need to inform decisions about what health services to adopt, use, or pay for? If we’re not sure, how will we engage our stakeholders to determine the priority metrics, incentives, or factors influencing decisions? Think about both short term and long term outcomes (refer to your logic model if you have created one).

7.3.2

What stakeholder-centered outcomes align with the key domains and/or constructs in our conceptual, theoretical, and/or process framework(s)?

7.3.3

What existing data sources are available to assess these outcomes? To what extent are these data sources pragmatic? That is, to what extent are these data collected in the course of routine practice (e.g., electronic health records), of high quality, and readily accessible to researchers? How will our outcomes be operationalized using these data sources?

References

  • Ross MK, Wei W, Ohno-Machado L. “Big data” and the electronic health record. Yearb Med Inform. 2014;9(1):97-104. Published 2014 Aug 15. doi:10.15265/IY-2014-0003
  • Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. 2017;106(1):1-9. doi:10.1007/s00392-016-1025-6
  • Sidebottom AC, Sillah A, Vock DM, et al. Assessing the impact of the heart of New Ulm Project on cardiovascular disease risk factors: A population-based program to reduce cardiovascular disease. Prev Med. 2018;112:216-221. doi:10.1016/j.ypmed.2018.04.016
  • Sidebottom AC, Sillah A, Miedema MD, et al. Changes in cardiovascular risk factors after 5 years of implementation of a population-based program to reduce cardiovascular disease: The Heart of New Ulm Project. Am Heart J. 2016;175:66-76. doi:10.1016/j.ahj.2016.02.006
  • Ru B, Yao L. A literature review of social media-based data mining for health outcomes research. In: Bian J, Guo Y, He Z, Hu X, editors. Social web and Health Research: benefits, limitations, and best practices. Cham: Springer International Publishing; 2019. pp. 1–14.
  • Taylor J, Pagliari C. Mining social media data: How are research sponsors and researchers addressing the ethical challenges? Research Ethics. 2017;14(2):1-39. doi:10.1177/1747016117738559

Videos

Mining and Analyzing Data from Social Media Data Sources

Planning Your Data Collection Strategies: Electronic Health Data

The Benefits and Challenges of Leveraging Existing and Secondary Data for Pragmatic Research: A Case Study of Evaluating the Effect of Living Kidney Donation on Long-Term Outcomes

Opportunities for Using Healthcare Claims Data for Pragmatic Sustainability Assessments

Digital Health Data Access, Management, and Use

7.3.4

For outcomes that require primary data collection, which types of data collection methods are most appropriate for which outcomes? Qualitative, quantitative, or mixed methods? How often is it feasible to collect repeated measures on a given setting or individual?

Videos

Planning Your Data Collection Strategies: Mixed Methods

Planning Your Data Collection Strategies: Rapid Qualitative Methods

7.4 Pragmatic Research Measures

Pragmatic measures are characterized by several key features:

  • Brevity
  • Criterion validity (does it predict what it is supposed to) 
  • Reliability (especially test-retest) 
  • Sensitivity to change (e.g. ability to detect intervention effects) 
  • Actionable and understood by users 
  • Quickly and easily scored 
  • Broad availability (e.g., validated in multiple languages and applicable across populations)
  • Availability of norms

It can be difficult to find measures that satisfy all of these criteria and align with frameworks and stakeholder priorities. There are also differences across sources, journals, and grant review sections on what constitutes appropriate pragmatic measures, such as the emphasis placed on internal consistency and other ‘traditional’ psychometric criteria as well as the extent to which a measure is face valid, actionable, and easily understood by users.

7.5 Key Resources for Pragmatic Research Measures

A variety of databases exist for identifying measures for pragmatic research. There are two specific resource repositories for pragmatic measures that specifically address how pragmatic a measure is along with information on its psychometric properties (GEM and SIRC).

Information on pragmatic measures is also often included among other more comprehensive sources of information on measures and evaluation procedures in general, such as the Buros mental measurement yearbook. Pragmatic sources of data include existing data sources such as electronic health records, claims data, national surveys (e.g., the Behavioral Risk Factor Surveillance Survey) and Census data.

References

Videos

Watson: Attics, Guesswork and Clay. Sleuthing Your Way into Biomedical Natural Language Processing

7.6 Pragmatic Research Measures – Key Questions

  • What valid, reliable self-report, survey, observational, interview, forced choice experiment, and/or behavioral measures are available? 
  • To what extent are these measures pragmatic? 
  • If we aren’t sure, what literature or measures databases will we review, or which experts will we consult with?

References

  • Blair G, Imai K, Zhou YY. Design and Analysis of the Randomized Response Technique. Journal of the American Statistical Association. 2015;110(511):1304-1319. doi:10.1080/01621459.2015.1050028
  • Fidler DS, Kleinknecht RE. Randomized response versus direct questioning: Two data-collection methods for sensitive information. Psychological Bulletin. 1977;84(5):1045-1049. doi:10.1037/0033-2909.84.5.1045
  • Ahart AM, Sackett PR. A New Method of Examining Relationships between Individual Difference Measures and Sensitive Behavior Criteria: Evaluating the Unmatched Count Technique. Organizational Research Methods. 2004;7(1):101-114. doi:10.1177/1094428103259557

Videos

Planning Your Data Collection Strategies: Behavioral Measures

7.7 Dissemination and Implementation Outcomes and Measures

References

  • Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT Figure: Increasing Transparency in Reporting on External Validity. Am J Prev Med. 2018;55(3):422-430. doi:10.1016/j.amepre.2018.04.044
  • Stanick C, Halko H, Mettert K, et al. Measuring characteristics of individuals: An updated systematic review of instruments’ psychometric properties. Implement Res Pract. 2021;2:26334895211000458. Published 2021 Mar 23. doi:10.1177/26334895211000458
  • Dorsey CN, Mettert KD, Puspitasari AJ, Damschroder LJ, Lewis CC. A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence. Implement Res Pract. 2021;2:26334895211002474. Published 2021 Apr 6. doi:10.1177/26334895211002474
  • Lewis CC, Mettert K, Lyon AR. Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review. Implement Res Pract. 2021;2:2633489521994197. Published 2021 Mar 23. doi:10.1177/2633489521994197
  • Weiner BJ, Mettert KD, Dorsey CN, et al. Measuring readiness for implementation: A systematic review of measures’ psychometric and pragmatic properties. Implement Res Pract. 2020;1:2633489520933896. Published 2020 Aug 26. doi:10.1177/2633489520933896
  • Mettert K, Lewis C, Dorsey C, Halko H, Weiner B. Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties. Implement Res Pract. 2020;1:2633489520936644. Published 2020 Aug 28. doi:10.1177/2633489520936644
  • McHugh S, Dorsey CN, Mettert K, Purtle J, Bruns E, Lewis CC. Measures of outer setting constructs for implementation research: A systematic review and analysis of psychometric quality. Implement Res Pract. 2020;1:2633489520940022. Published 2020 Jul 21. doi:10.1177/2633489520940022

Videos

Measuring Implementation Outcomes

Assessing and Enhancing Reach and Representativeness