Section 5: Pragmatic Study Design

5.1 Pragmatic Study Design – Key Questions

Use the following questions to help determine appropriate study design types for your pragmatic research. When we refer to an intervention, we mean any program, treatment, service, or policy that will be tested in the setting in which it is intended to be used or delivered. Contact a biostatistician (and possibly other experts such as health economist, qualitative analysis expert, social network or systems analyst) early to discuss appropriate study designs and analytic techniques. 

Will my design type be:

References

  • Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. Published 2015 May 8. doi:10.1136/bmj.h2147
  • Kutner, J. S., Blatchford, P. J., Taylor, D. H., Ritchie, C. S., Bull, J. H., Fairclough, D. L., … & Abernethy, A. P. (2015). Safety and benefit of discontinuing statin therapy in the setting of advanced, life-limiting illness: a randomized clinical trial. JAMA internal medicine, 175(5), 691-700. doi:10.1001/jamainternmed.2015.0289

5.1.1

Participant-level randomized trial?

References

5.1.2

Cluster randomized trial – at what level of randomization/outcome data?

References

  • Cook AJ, Delong E, Murray DM, Vollmer WM, Heagerty PJ. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials. 2016;13(5):504-512. doi:10.1177/1740774516646578

Videos

Cluster Randomized Trials and a Case Example

5.1.3

Stepped wedge design – at what level of randomization at rollout?

References

  • Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391. Published 2015 Feb 6. doi:10.1136/bmj.h391need link
  • Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28(2):182-191. doi:10.1016/j.cct.2006.05.007
  • Hemming K, Lilford R, Girling AJ. Stepped-wedge cluster randomised controlled trials: a generic framework including parallel and multiple-level designs. Stat Med. 2015;34(2):181-196. doi:10.1002/sim.6325

Videos

Recent Developments in Statistical Methods for Pragmatic, Stepped Wedge Cluster Randomized Trials

Stepped Wedge Design in Practice: A Checklist for Feasibility

5.1.4

Quasi-experimental design – what type?

References

  • Handley MA, Lyles CR, McCulloch C, Cattamanchi A. Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research. Annu Rev Public Health. 2018;39:5-25. doi:10.1146/annurev-publhealth-040617-014128
  • Kontopantelis E, Doran T, Springate DA, Buchan I, Reeves D. Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ. 2015;350:h2750. Published 2015 Jun 9. doi:10.1136/bmj.h2750
  • Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial [published correction appears in Int J Epidemiol. 2020 Aug 1;49(4):1414. doi: 10.1093/ije/dyaa118.]. Int J Epidemiol. 2017;46(1):348-355. doi:10.1093/ije/dyw098
  • Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013;13(6 Suppl):S38-S44. doi:10.1016/j.acap.2013.08.002
  • Lockwood J, Moss A, Beck A, Francis I, Schmoll E, Wymore E. The association between the legalization of recreational marijuana and both small for gestational age births and NICU admissions in Colorado. J Perinatol. 2019;39(9):1165-1174. doi:10.1038/s41372-019-0416-8

5.1.5

Observational design – what type?

References

  • Hripcsak G, Schuemie MJ, Madigan D, Ryan PB, Suchard MA. Drawing Reproducible Conclusions from Observational Clinical Data with OHDSI. Yearb Med Inform. 2021;30(1):283-289. doi:10.1055/s-0041-1726481
  • Felmeister AS, Waanders AJ, Leary SE, et al. Preliminary exploratory data analysis of simulated National Clinical Data Research Network for future use in annotation of a rare tumor biobanking initiative. 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Published online November 2017. doi:10.1109/bibm.2017.8217983
  • Zubizarreta JR, Small DS, Rosenbaum PR. Isolation in the construction of natural experiments. The Annals of Applied Statistics. 2014;8(4). doi:10.1214/14-aoas770
  • Czajkowski SM, Powell LH, Adler N, et al. From ideas to efficacy: The ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971-982. doi:10.1037/hea0000161
  • Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning Clinical Science: Unifying the Discipline to Improve the Public Health. Clin Psychol Sci. 2014;2(1):22-34. doi:10.1177/2167702613497932
  • Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016;183(8):758-764. doi:10.1093/aje/kwv254
  • Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Stat Med. 2014;33(13):2297-2340. doi:10.1002/sim.6128

Videos

Natural Experiments and Observational Studies

5.1.6

Factorial (full or partial) design?

References

  • Kahan BC, Hall SS, Beller EM, et al. Reporting of Factorial Randomized Trials: Extension of the CONSORT 2010 Statement. JAMA. 2023;330(21):2106-2114. doi:10.1001/jama.2023.19793need link
  • Berlin JA. What Are Factorial Experiments and Why Can They Be Helpful?. JAMA Netw Open. 2019;2(9):e1911917. Published 2019 Sep 4. doi:10.1001/jamanetworkopen.2019.11917
  • Brown JC, Sarwer DB, Troxel AB, et al. A randomized trial of exercise and diet on health-related quality of life in survivors of breast cancer with overweight or obesity. Cancer. 2021;127(20):3856-3864. doi:10.1002/cncr.33752

5.1.7

SMART design?

References

  • Almirall D, Nahum-Shani I, Sherwood NE, Murphy SA. Introduction to SMART designs for the development of adaptive interventions: with application to weight loss research. Transl Behav Med. 2014;4(3):260-274. doi:10.1007/s13142-014-0265-0
  • Rush AJ, Fava M, Wisniewski SR, et al. Sequenced treatment alternatives to relieve depression (STAR*D): rationale and design. Control Clin Trials. 2004;25(1):119-142. doi:10.1016/s0197-2456(03)00112-0
  • Pfammatter AF, Nahum-Shani I, DeZelar M, et al. SMART: Study protocol for a sequential multiple assignment randomized controlled trial to optimize weight loss management. Contemp Clin Trials. 2019;82:36-45. doi:10.1016/j.cct.2019.05.007
  • Kelleher SA, Dorfman CS, Plumb Vilardaga JC, et al. Optimizing delivery of a behavioral pain intervention in cancer patients using a sequential multiple assignment randomized trial SMART. Contemp Clin Trials. 2017;57:51-57. doi:10.1016/j.cct.2017.04.001
  • Wolbers M, Helterbrand JD. Two-stage randomization designs in drug development. Stat Med. 2008;27(21):4161-4174. doi:10.1002/sim.3309
  • Pistorello J, Jobes DA, Compton SN, et al. Developing Adaptive Treatment Strategies to Address Suicidal Risk in College Students: A Pilot Sequential, Multiple Assignment, Randomized Trial (SMART). Arch Suicide Res. 2017;22(4):644-664. doi:10.1080/13811118.2017.1392915
  • Kidwell KM. SMART designs in cancer research: Past, present, and future. Clin Trials. 2014;11(4):445-456. doi:10.1177/1740774514525691
  • Lei H, Nahum-Shani I, Lynch K, Oslin D, Murphy SA. A “SMART” design for building individualized treatment sequences. Annu Rev Clin Psychol. 2012;8:21-48. doi:10.1146/annurev-clinpsy-032511-143152
  • Nahum-Shani I, Qian M, Almirall D, et al. Experimental design and primary data analysis methods for comparing adaptive interventions. Psychol Methods. 2012;17(4):457-477. doi:10.1037/a0029372

Videos

A Tour of Pragmatic Study Design: SMART Trials

5.1.8

Adaptive design?

References

  • Thorlund K, Haggstrom J, Park JJ, Mills EJ. Key design considerations for adaptive clinical trials: a primer for clinicians. BMJ. 2018;360:k698. Published 2018 Mar 8. doi:10.1136/bmj.k698
  • Chow SC. Adaptive clinical trial design. Annu Rev Med. 2014;65:405-415. doi:10.1146/annurev-med-092012-112310
  • Li Q, Lin J, Lin Y. Adaptive design implementation in confirmatory trials: methods, practical considerations and case studies. Contemp Clin Trials. 2020;98:106096. doi:10.1016/j.cct.2020.106096
  • Kairalla JA, Coffey CS, Thomann MA, Muller KE. Adaptive trial designs: a review of barriers and opportunities. Trials. 2012;13:145. Published 2012 Aug 23. doi:10.1186/1745-6215-13-145
  • Kaizer AM, Belli HM, Ma Z, et al. Recent innovations in adaptive trial designs: A review of design opportunities in translational research. J Clin Transl Sci. 2023;7(1):e125. Published 2023 Apr 28. doi:10.1017/cts.2023.537

5.2 Randomization

Is randomization to condition possible, ethical, and feasible? Why or why not? 

5.2.1

For non-randomized designs, consider an observational, quasi-experimental design, or natural experiment.

References

    • Zubizarreta JR, Small DS, Rosenbaum PR. Isolation in the construction of natural experiments. The Annals of Applied Statistics. 2014;8(4). doi:10.1214/14-aoas770
    • Czajkowski SM, Powell LH, Adler N, et al. From ideas to efficacy: The ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971-982. doi:10.1037/hea0000161
    • Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning Clinical Science: Unifying the Discipline to Improve the Public Health. Clin Psychol Sci. 2014;2(1):22-34. doi:10.1177/2167702613497932
    • Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. Am J Epidemiol. 2016;183(8):758-764. doi:10.1093/aje/kwv254
    • Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Stat Med. 2014;33(13):2297-2340. doi:10.1002/sim.6128

5.2.2

For randomized designs:

  • Will randomization be at the participant level or provider/site/cluster level? Why? Consider a cluster randomized trial or stepped wedge design if there is the possibility of contamination or pragmatic challenges with participant level randomization (e.g., an organization or provider would be unable to deliver an intervention more than one way at a time due to resources)
  • Is the recruitment rate likely to be constant across time? If no, consider cluster randomized rather than stepped wedge to mitigate study delays when recruitment is low.
  • How feasible is it to implement the intervention for all randomization units at the same time? If not feasible, consider a stepped wedge to distribute the implementation at clusters at different time points.
  • Are more than two interventions being compared? If yes, consider a cluster randomized trial or a participant-level randomized trial instead of a stepped wedge design.
  • Do the intervention(s) to be tested have multiple components that need to be optimized in terms of combination, sequence, dose, or tailoring? If yes, an adaptive trial design (e.g., SMART) or factorial design may be appropriate. Also considered a MOST approach for iterative design and testing of an optimized intervention strategy.

References

  • Guastaferro K, Collins LM. Achieving the Goals of Translational Science in Public Health Intervention Research: The Multiphase Optimization Strategy (MOST). Am J Public Health. 2019;109(S2):S128-S129. doi:10.2105/AJPH.2018.304874
  • Collins LM, Springerlink (Online Service. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions : The Multiphase Optimization Strategy (MOST). Springer International Publishing; 2018.
  • DeBar L, Benes L, Bonifay A, et al. Interdisciplinary team-based care for patients with chronic pain on long-term opioid treatment in primary care (PPACT) – Protocol for a pragmatic cluster randomized trial. Contemp Clin Trials. 2018;67:91-99. doi:10.1016/j.cct.2018.02.015
  • Caille A, Kerry S, Tavernier E, Leyrat C, Eldridge S, Giraudeau B. Timeline cluster: a graphical tool to identify risk of bias in cluster randomised trials. BMJ. 2016;354:i4291. Published 2016 Aug 16. doi:10.1136/bmj.i4291

5.3 Power and Sample Size Estimation

Pragmatic trials with an active comparator may anticipate a small effect size difference, which requires more participants to achieve adequate statistical power. What is your anticipated effect size difference for your study? Do you have access to the required sample size in your partnering sites? 

References

  • Samsa GP, Winger JG, Cox CE, Olsen MK. Two Questions About the Design of Cluster Randomized Trials: A Tutorial. J Pain Symptom Manage. 2021;61(4):858-863. doi:10.1016/j.jpainsymman.2020.11.019

5.4 Analysis

Standard methods for analysis of individually randomized trials may not be appropriate. Statistical analysis must incorporate the study design features, such as hierarchical dependency of data and temporal trends. What analytic approach(es) might be appropriate? Contact a biostatistician (and possibly other experts such as a health economist, qualitative analysis expert, social network or systems analyst) early to discuss appropriate study designs and analytic techniques.

References

  • Steyerberg E. Clinical Prediction Models : A Practical Approach to Development, Validation and Updating. Springer; 2019.
  • Dess RT, Suresh K, Zelefsky MJ, et al. Development and Validation of a Clinical Prognostic Stage Group System for Nonmetastatic Prostate Cancer Using Disease-Specific Mortality Results From the International Staging Collaboration for Cancer of the Prostate. JAMA Oncol. 2020;6(12):1912-1920. doi:10.1001/jamaoncol.2020.4922
  • Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1-W73. doi:10.7326/M14-0698
  • Samsa G, Colborn K, Olsen M, et al. A Visual Tool to Help Develop a Statistical Analysis Plan for Randomized Trials in Palliative Care. J Pain Symptom Manage. 2023;65(1):e87-e95. doi:10.1016/j.jpainsymman.2022.08.004
  • Gamble C, Krishan A, Stocken D, et al. Guidelines for the Content of Statistical Analysis Plans in Clinical Trials. JAMA. 2017;318(23):2337-2343. doi:10.1001/jama.2017.18556
  • Ali SA, Kloseck M, Lee K, Walsh KE, MacDermid JC, Fitzsimmons D. Evaluating the design and reporting of pragmatic trials in osteoarthritis research. Rheumatology (Oxford). 2018;57(1):59-63. doi:10.1093/rheumatology/kex050