Section 5: Pragmatic Study Design

5.1 Pragmatic Study Design – Key Questions

Use the following questions to help determine appropriate study design types for your pragmatic research. When we refer to an intervention, we mean any program, treatment, service, or policy that will be tested in the setting in which it is intended to be used or delivered. Contact a biostatistician (and possibly other experts such as health economist, qualitative analysis expert, social network or systems analyst) early to discuss appropriate study designs and analytic techniques. 

Will my design type be:

References

  • Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. Published 2015 May 8. doi:10.1136/bmj.h2147

5.1.1

Participant-level randomized trial?

References

  • [Example publications for design – need example]. need link

5.1.2

Cluster randomized trial – at what level of randomization/outcome data?

References

  • Cook AJ, Delong E, Murray DM, Vollmer WM, Heagerty PJ. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials. 2016;13(5):504-512. doi:10.1177/1740774516646578

Videos

Cluster Randomized Trials and a Case Example

5.1.3

Stepped wedge design – at what level of randomization at rollout?

References

  • [Example publications for design- NEED EXAMPLE]. need link

Videos

Recent Developments in Statistical Methods for Pragmatic, Stepped Wedge Cluster Randomized Trials

Stepped Wedge Design in Practice: A Checklist for Feasibility

5.1.4

Quasi-experimental design – what type?

References

  • [Example publications for design- NEED EXAMPLE]. need link

5.1.5

Observational design – what type?

References

  • Hripcsak G, Schuemie MJ, Madigan D, Ryan PB, Suchard MA. Drawing Reproducible Conclusions from Observational Clinical Data with OHDSI. Yearb Med Inform. 2021;30(1):283-289. doi:10.1055/s-0041-1726481
  • Felmeister AS, Waanders AJ, Leary SE, et al. Preliminary exploratory data analysis of simulated National Clinical Data Research Network for future use in annotation of a rare tumor biobanking initiative. 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Published online November 2017. doi:10.1109/bibm.2017.8217983
  • Zubizarreta JR, Small DS, Rosenbaum PR. Isolation in the construction of natural experiments. The Annals of Applied Statistics. 2014;8(4). doi:10.1214/14-aoas770
  • Czajkowski SM, Powell LH, Adler N, et al. From ideas to efficacy: The ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971-982. doi:10.1037/hea0000161
  • Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning Clinical Science: Unifying the Discipline to Improve the Public Health. Clin Psychol Sci. 2014;2(1):22-34. doi:10.1177/2167702613497932
  • Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016;183(8):758-764. doi:10.1093/aje/kwv254
  • Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Stat Med. 2014;33(13):2297-2340. doi:10.1002/sim.6128

Videos

Natural Experiments and Observational Studies

5.1.6

Factorial (full or partial) design?

References

  • [Example publications for design- NEED EXAMPLE]. need link

5.1.7

SMART design?

References

  • Almirall D, Nahum-Shani I, Sherwood NE, Murphy SA. Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research. Transl Behav Med. 2014;4(3):260-274. doi:10.1007/s13142-014-0265-0
  • Rush AJ, Fava M, Wisniewski SR, et al. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25(1):119-142. doi:10.1016/s0197-2456(03)00112-0
  • Pfammatter AF, Nahum-Shani I, DeZelar M, et al. SMART: Study protocol for a sequential multiple assignment randomized controlled trial to optimize weight loss management. Contemp Clin Trials. 2019;82:36-45. doi:10.1016/j.cct.2019.05.007
  • Kelleher SA, Dorfman CS, Plumb Vilardaga JC, et al. Optimizing delivery of a behavioral pain intervention in cancer patients using a sequential multiple assignment randomized trial SMART. Contemp Clin Trials. 2017;57:51-57. doi:10.1016/j.cct.2017.04.001
  • Wolbers M, Helterbrand JD. Two-stage randomization designs in drug development. Stat Med. 2008;27(21):4161-4174. doi:10.1002/sim.3309
  • Pistorello J, Jobes DA, Compton SN, et al. Developing Adaptive Treatment Strategies to Address Suicidal Risk in College Students: A Pilot Sequential, Multiple Assignment, Randomized Trial (SMART). Arch Suicide Res. 2017;22(4):644-664. doi:10.1080/13811118.2017.1392915
  • Kidwell KM. SMART designs in cancer research: Past, present, and future. Clin Trials. 2014;11(4):445-456. doi:10.1177/1740774514525691
  • Lei H, Nahum-Shani I, Lynch K, Oslin D, Murphy SA. A “SMART” design for building individualized treatment sequences. Annu Rev Clin Psychol. 2012;8:21-48. doi:10.1146/annurev-clinpsy-032511-143152
  • Nahum-Shani I, Qian M, Almirall D, et al. Experimental design and primary data analysis methods for comparing adaptive interventions. Psychol Methods. 2012;17(4):457-477. doi:10.1037/a0029372

Videos

A Tour of Pragmatic Study Design: SMART Trials

5.1.8

Adaptive design?

References

  • [Example publications for design- NEED EXAMPLE]. need link

5.2 Randomization

Is randomization to condition possible, ethical, and feasible? Why or why not? 

5.2.1

For non-randomized designs, consider an observational, quasi-experimental design, or natural experiment.

References

    • Zubizarreta JR, Small DS, Rosenbaum PR. Isolation in the construction of natural experiments. The Annals of Applied Statistics. 2014;8(4). doi:10.1214/14-aoas770
    • Czajkowski SM, Powell LH, Adler N, et al. From ideas to efficacy: The ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971-982. doi:10.1037/hea0000161
    • Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning Clinical Science: Unifying the Discipline to Improve the Public Health. Clin Psychol Sci. 2014;2(1):22-34. doi:10.1177/2167702613497932
    • Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016;183(8):758-764. doi:10.1093/aje/kwv254
    • Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Stat Med. 2014;33(13):2297-2340. doi:10.1002/sim.6128

5.2.2

For randomized designs:

  • Will randomization be at the participant level or provider/site/cluster level? Why? Consider a cluster randomized trial or stepped wedge design if there is the possibility of contamination or pragmatic challenges with participant level randomization (e.g., an organization or provider would be unable to deliver an intervention more than one way at a time due to resources)
  • Is the recruitment rate likely to be constant across time? If no, consider cluster randomized rather than stepped wedge to mitigate study delays when recruitment is low.
  • How feasible is it to implement the intervention for all randomization units at the same time? If not feasible, consider a stepped wedge to distribute the implementation at clusters at different time points. o Are more than two interventions being compared? If yes, consider a cluster randomized trial or a participant-level randomized trial instead of a stepped wedge design.
  • Do the intervention(s) to be tested have multiple components that need to be optimized in terms of combination, sequence, dose, or tailoring? If yes, an adaptive trial design (e.g., SMART) or factorial design may be appropriate. Also considered a MOST approach for iterative design and testing of an optimized intervention strategy.

References

  • Guastaferro K, Collins LM. Achieving the Goals of Translational Science in Public Health Intervention Research: The Multiphase Optimization Strategy (MOST). Am J Public Health. 2019;109(S2):S128-S129. doi:10.2105/AJPH.2018.304874
  • Collins LM, Springerlink (Online Service. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions : The Multiphase Optimization Strategy (MOST). Springer International Publishing; 2018.
  • DeBar L, Benes L, Bonifay A, et al. Interdisciplinary team-based care for patients with chronic pain on long-term opioid treatment in primary care (PPACT) – Protocol for a pragmatic cluster randomized trial. Contemp Clin Trials. 2018;67:91-99. doi:10.1016/j.cct.2018.02.015
  • Caille A, Kerry S, Tavernier E, Leyrat C, Eldridge S, Giraudeau B. Timeline cluster: a graphical tool to identify risk of bias in cluster randomised trials. BMJ. 2016;354:i4291. Published 2016 Aug 16. doi:10.1136/bmj.i4291

5.3 Power and Sample Size Estimation

Pragmatic trials with an active comparator may anticipate a small effect size difference, which requires more participants to achieve adequate statistical power. What is your anticipated effect size difference for your study? Do you have access to the required sample size in your partnering sites? 

5.4 Analysis

Standard methods for analysis of individually randomized trials may not be appropriate. Statistical analysis must incorporate the study design features, such as hierarchical dependency of data and temporal trends. What analytic approach(es) might be appropriate? Contact a biostatistician (and possibly other experts such as a health economist, qualitative analysis expert, social network or systems analyst) early to discuss appropriate study designs and analytic techniques.

References

  • Steyerberg E. Clinical Prediction Models : A Practical Approach to Development, Validation and Updating. Springer; 2019.
  • Dess RT, Suresh K, Zelefsky MJ, et al. Development and Validation of a Clinical Prognostic Stage Group System for Nonmetastatic Prostate Cancer Using Disease-Specific Mortality Results From the International Staging Collaboration for Cancer of the Prostate. JAMA Oncol. 2020;6(12):1912-1920. doi:10.1001/jamaoncol.2020.4922
  • Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-W73. doi:10.7326/M14-0698