Date of Award

January 2026

Document Type

Open Access Thesis

Degree Name

Medical Doctor (MD)

Department

Medicine

First Advisor

Joseph S. Ross

Abstract

BackgroundClinicians rely on clinical practice guidelines (CPGs) to fulfill a dual mandate: practicing evidence-based medicine and ensuring cost-conscious stewardship of resources. However, it remains unclear whether modern CPGs provide the guidance necessary to meet these goals. Specifically, the extent to which recommendation strengths align with evidence quality and the prevalence of cost-related guidance within U.S. medical society CPGs remains unknown.

Research Aimsa) To examine the distribution of SOR and LOE ratings for recommendations offered by United States medical societies through CPGs and evaluate concordance between SOR and LOE ratings for CPG recommendations. Additionally, assess whether there are recommendation-level, guideline-level, or society-level characteristics which affect SOR-LOE rating concordance among CPG recommendations. b) To characterize the proportion of CPG recommendations with cost-related considerations. Additionally, assess whether there are recommendation-level, guideline-level, or society-level characteristics associated with recommendations including cost-related considerations.

Research Questionsa) To what extent are SOR ratings for CPG recommendations informed by equivalent or stronger LOE ratings? Which recommendation-level, guideline-level, or society-level characteristics affect SOR-LOE rating concordance? b) What proportion of CPG recommendations include information pertaining to cost-related considerations? Which recommendation-level, guideline-level, or society-level characteristics affect the likelihood that cost-related considerations are included?

MethodsCross-sectional analysis of recommendations from 309 CPG documents issued by 23 United States-based medical societies from January 1, 2019, to December 31, 2023. Main measures for analyses pertaining to SOR-LOE rating concordance: Recommendation-level variables: SOR and LOE ratings. Society-level variables: methodology for assigning SOR and LOE ratings and society’s specialty type. SOR-LOE ratings were concordant if LOE rating was at least as strong as SOR rating, with concordance quantified using modified Cohen’s kappa []. Main measures for analyses pertaining to cost-related considerations: Recommendation-level characteristics: intervention type. Guideline-level characteristics: pediatric focus and multi-society consensus effort. Society-level characteristics: primary-care versus specialty-care society and a society’s methodological approach to considering cost-related evidence. For each recommendation, the presence of cost-related considerations in (1) the upfront text summarizing the society’s position on an aspect of clinical care and (2) the narrative discussion of the supporting evidence was determined.

ResultsFor analyses pertaining to SOR-LOE concordance: Among 7,582 recommendations, society guideline writing committees rated SOR as strong for 2,303 (30%) recommendations, moderate for 1,351 (18%), weak for 2,028 (27%), and 1,900 recommendations (25%) had no strength assigned. Regarding LOE ratings, 768 (10%) recommendations had strong evidence, 1,676 (22%) had moderate evidence, 2,421 (32%) had weak or very weak evidence, and 2,717 (36%) had evidence lacking an LOE rating. Overall, 5,088 (67.1%) recommendations had SOR-LOE rating concordance, and overall SOR-LOE rating concordance was moderate (κ=0.67; 95% confidence interval [CI] 0.66-0.68). Strong recommendations, however, had minimal SOR-LOE concordance (κ=0.27; 95% CI 0.25-0.28; p=<0.001), with 1,690 (73%) having LOE ratings weaker than strong. Recommendations based on a Modified-Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) Framework (κ=0.33; 95% CI 0.30-0.37, p<0.001), and issued by obstetrics/gynecology (κ=0.46; 95% CI 0.40-0.51, p<0.001) or neurology/psychiatry (κ=0.22; 95% CI 0.18-0.26, p<0.001) societies had significantly lower rating concordance. For analyses pertaining to cost-related considerations: There were 309 CPG documents containing 7,582 recommendations, of which 4,080 (53.8%) were issued through multi-society consensus efforts and 286 (3.8%) by primary-care societies. In examining the 23 societies’ methodology guides for developing CPGs, 14 (61%) societies explicitly consider costs, 6 (26%) ambiguously consider costs, and 3 (13%) never mention costs. Overall, 287 (3.8%) recommendations contained upfront cost-related statements, and 1,706 (22.5%) narratively discussed cost-related evidence. In multi-level, multivariable analyses, upfront cost-related statements were more likely if recommendations were issued by a primary care society (OR 8.09; 95% CI 1.90-34.56; p=0.005) or were treatment-related (OR 1.76; 95% CI 1.19-2.62; p=0.005) and less likely if developed by multi-society consensus (OR 0.16; 95% CI 0.06-0.45; p=0.001). Narrative cost-related evidence discussions were more likely if the recommendations were issued by a society that explicitly considered costs (OR 5.72; 95% CI 2.10-15.61; p=0.001) and less likely if pediatric-focused (OR 0.30; 95% CI 0.10-0.91; p=0.03) or formulated by multi-society consensus effort (OR 0.49; 95% CI 0.27-0.89; p=0.02). Other characteristics were not associated with recommendations including cost-related considerations.

SignificanceAmong CPG recommendations examined, overall SOR-LOE rating concordance was moderate, although recommendations with strong SOR ratings had minimal rating concordance. CPG authors could provide clearer rationales for strong recommendations when the corresponding evidence does not reflect commensurate certainty. Although three-in-five medical societies examined report explicitly considering cost-related evidence when issuing CPG recommendations, cost-related considerations are infrequently mentioned, which may hinder clinicians’ ability to practice evidence-based, cost-conscious care.

Comments

This is an Open Access Thesis.

Download
Open Access

This Article is Open Access

Share

COinS