Advertisement

Mandatory public reporting of cardiac surgery outcomes: The 2003 to 2014 Massachusetts experience

Open ArchivePublished:December 31, 2018DOI:https://doi.org/10.1016/j.jtcvs.2018.12.072

      Abstract

      Objectives

      Beginning in 2002, all 14 Massachusetts nonfederal cardiac surgery programs submitted Society of Thoracic Surgeons (STS) National Database data to the Massachusetts Data Analysis Center for mandatory state-based analysis and reporting, and to STS for nationally benchmarked analyses. We sought to determine whether longitudinal prevalences and trends in risk factors and observed and expected mortality differed between Massachusetts and the nation.

      Methods

      We analyzed 2003 to 2014 expected (STS predicted risk of operative [in-hospital + 30-day] mortality), observed, and risk-standardized isolated coronary artery bypass graft mortality using Massachusetts STS data (N = 39,400 cases) and national STS data (N = 1,815,234 cases). Analyses included percentage shares of total Massachusetts coronary artery bypass graft volume and expected mortality rates of 2 hospitals before and after outlier designation.

      Results

      Massachusetts patients had significantly higher odds of diabetes, peripheral vascular disease, low ejection fraction, and age ≥75 years relative to national data and lower odds of shock (odds ratio, 0.66; 99% confidence interval, 0.53-0.83), emergency (odds ratio, 0.57, 99% confidence interval, 0.52-0.61), reoperation, chronic lung disease, dialysis, obesity, and female sex. STS predicted risk of operative [in-hospital + 30-day] mortality for Massachusetts patients was higher than national rates during 2003 to 2007 (P < .001) and no different during 2008 to 2014 (P = .135). Adjusting for STS predicted risk of operative [in-hospital + 30-day] mortality, Massachusetts patients had significantly lower odds (odds ratio, 0.79; 99% confidence interval, 0.66-0.96) of 30-day mortality relative to national data. Outlier programs experienced inconsistent, transient influences on expected mortality and their percentage shares of Massachusetts coronary artery bypass graft cases.

      Conclusions

      During 12 years of mandatory public reporting, Massachusetts risk-standardized coronary artery bypass graft mortality was consistently and significantly lower than national rates, expected rates were comparable or higher, and evidence for risk aversion was conflicting and inconclusive.

      Key Words

      Abbreviations and Acronyms:

      ACSD (Society of Thoracic Surgeons Adult Cardiac Surgery Database), CABG (coronary artery bypass grafting), CY (calendar year), DPH (Department of Public Health), EF (ejection fraction), FY (fiscal year), Mass-DAC (Massachusetts Data Analysis Center), O/E (observed to expected ratio), PCI (percutaneous coronary intervention), PVD (peripheral vascular disease), STS-PROM (Society of Thoracic Surgeons predicted risk of mortality)
      Figure thumbnail fx1
      Operative mortality, Society of Thoracic Surgeons (STS) national data (excluding Massachusetts) versus STS data exclusively from Massachusetts and Massachusetts Data Analysis Center (Mass-DAC).
      From 2003-2014, risk-adjusted CABG outcomes in Massachusetts, a public reporting state, were consistently superior to STS national results; evidence for risk aversion was conflicting and inconclusive.
      During 12 years of mandatory public reporting (2003-2014), Massachusetts risk-adjusted CABG mortality was consistently lower than national rates, expected mortality was comparable or higher, and evidence for risk aversion was inconclusive. These nationally benchmarked results support continued study of public reporting as a mechanism to educate patients and potentially to improve health care outcomes.
      See Commentary on page 125.
      Public reporting of hospital-specific outcomes by government and commercial organizations remains highly controversial. Proponents cite the need for transparency and accountability, the ethical obligation to facilitate informed decision making by consumers and referring physicians, and the ability of report cards to stimulate performance improvement. Detractors are concerned about the accuracy and effectiveness of report cards and their unintended consequences such as gaming, risk aversion, and reduced access to care.
      • Shahian D.M.
      • Edwards F.H.
      • Jacobs J.P.
      • Prager R.L.
      • Normand S.L.
      • Shewan C.M.
      • et al.
      Public reporting of cardiac surgery performance: part 1-history, rationale, consequences.
      • Shahian D.M.
      • Edwards F.H.
      • Jacobs J.P.
      • Prager R.L.
      • Normand S.L.
      • Shewan C.M.
      • et al.
      Public reporting of cardiac surgery performance: part 2-implementation.
      • Shahian D.M.
      • Normand S.L.
      • Friedberg M.W.
      • Hutter M.M.
      • Pronovost P.J.
      Rating the raters: the inconsistent quality of health care performance measurement.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. part 1: observations from cardiac surgery and interventional cardiology.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. Part 2: mitigation strategies.
      • Hannan E.L.
      • Cozzens K.
      • King III, S.B.
      • Walford G.
      • Shah N.R.
      The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes.
      • Schneider E.C.
      • Epstein A.M.
      Use of public performance reports: a survey of patients undergoing cardiac surgery.
      • Schneider E.C.
      • Epstein A.M.
      Influence of cardiac-surgery performance reports on referral practices and access to care—a survey of cardiovascular specialists.
      • Harlan B.J.
      Statewide reporting of coronary artery surgery results: a view from California.
      • Shahian D.M.
      • Grover F.L.
      • Prager R.L.
      • Edwards F.H.
      • Filardo G.
      • O'Brien S.M.
      • et al.
      The Society of Thoracic Surgeons voluntary public reporting initiative: the first 4 years.
      The Massachusetts cardiac surgery public reporting program provides a unique natural laboratory in which to study these issues. State legislation mandates the use of the Society of Thoracic Surgeons (STS) Adult Cardiac Surgery Database (ACSD) by all cardiac surgery programs, which ensures a highly representative, audited, and risk-adjusted national comparator for Massachusetts, using identical data specifications.
      Beginning January 2002, Massachusetts Department of Public Health (DPH) required collection of clinical and vital status data from all 14 nonfederal Massachusetts cardiac surgery programs, stipulating participation in the STS ACSD
      • Shahian D.M.
      • Jacobs J.P.
      • Edwards F.H.
      • Brennan J.M.
      • Dokholyan R.S.
      • Prager R.L.
      • et al.
      The Society of Thoracic Surgeons National database.
      • Shahian D.M.
      • Jacobs J.P.
      Health services information: lessons learned from the Society of Thoracic Surgeons National database.
      and parallel submission of fully identified data to the Massachusetts Data Analysis Center (Mass-DAC). Reports of hospital outcomes for isolated coronary artery bypass grafting (CABG) were published for 2002 to 2014 procedures, and surgeon outcomes for 2002 to 2010 procedures.
      Massachusett Department of Health
      Cardiac services and treatments.
      Because of common STS ACSD data platforms, cross-sectional and longitudinal analyses of cardiac surgical practice in Massachusetts versus STS national data are facilitated. We leverage this to compare Massachusetts and STS isolated CABG volumes; expected, observed, and risk-standardized outcomes; and risk factor prevalences. The value of peer coding adjudication in Massachusetts and the consequences of outlier designation were also studied.

      Methods

      Study Populations

      We excluded 2002 data due to inaugural year coding issues, similar to early New York reports.
      • Green J.
      • Wintfeld N.
      Report cards on cardiac surgeons. Assessing New York State's approach.
      • Chassin M.R.
      • Hannan E.L.
      • DeBuono B.A.
      Benefits and hazards of reporting medical outcomes publicly.
      We analyzed 3 populations of adults aged 18 years or older undergoing isolated CABG between January 1, 2003, and December 31, 2014: isolated CABG cases submitted to STS from Massachusetts hospitals (MA [STS] cohort, n = 39,400), isolated CABG procedures submitted to STS from US states, excluding Massachusetts (national [STS] cohort, n = 1,815,234), and isolated CABG cases in all nonfederal Massachusetts hospitals, submitted to Mass-DAC (n = 40,655) for the Massachusetts public reporting program. The additional Mass-DAC cases compared with cases submitted to STS reflects slight differences in isolated CABG case inclusion criteria between the 2 sources.

      Data Sources

      We used information publicly reported on the Massachusetts DPH
      Massachusett Department of Health
      Cardiac services and treatments.
      and STS websites,
      The Society of Thoracic Surgeons
      Homepage.
      previously published Mass-DAC reports, and additional data provided by these organizations.

      Patient and Hospital Outcomes

      Mass-DAC reports observed, expected, and risk-standardized 30-day all-cause mortality. STS reports observed and expected operative mortality, which includes all deaths during the index hospitalization, regardless of timing, and all deaths within 30 days regardless of location. In addition to these standard reports, we extracted STS 30-day observed mortality and Mass-DAC operative mortality from original data sources.

      Data Collection and Quality Assurance

      STS cohort

      STS ACSD data specifications, cleaning, and quality assurance processes are published elsewhere.
      • Shahian D.M.
      • Jacobs J.P.
      • Edwards F.H.
      • Brennan J.M.
      • Dokholyan R.S.
      • Prager R.L.
      • et al.
      The Society of Thoracic Surgeons National database.
      • Shahian D.M.
      • Jacobs J.P.
      Health services information: lessons learned from the Society of Thoracic Surgeons National database.
      The STS National database.
      Annual external audits of 10% of participating programs demonstrate 95% to 97% overall agreement rates. During the study period, multiple STS ACSD versions (2.35-2.81) were used depending on year.
      Data submitted to STS were missing <1% for most variables, presumably because Massachusetts data managers were required to have complete data for their contemporaneous Mass-DAC submissions. Patients with missing or unknown operative mortality status were assumed alive, consistent with STS coding practice during the study period.

      Mass-DAC cohort

      Mass-DAC data collection, cleaning, and quality assurance processes are published elsewhere.
      Massachusett Department of Health
      Cardiac services and treatments.
      Massachusetts Data Analysis Center
      Adult coronary artery bypass graft surgery FY14 annual report.
      Mass-DAC reports information assembled from several linked data sources. Patient-specific risk factor and outcome data are collected by trained hospital personnel using standard STS ACSD data submission forms.
      The STS National database.
      Massachusetts Data Analysis Center
      Adult coronary artery bypass graft surgery FY14 annual report.
      External administrative data resources and medical record review are used to validate submitted data and to ascertain or verify 30-day vital status (Table E1). Missing data were extremely rare.
      Before analyses, Mass-DAC charts coded for selected risk factors (routinely: shock, emergent or emergent/salvage status, myocardial infarction <24 hours, and, beginning 2012, peripheral vascular disease [PVD]; periodically, low ejection fraction [EF], moderate/severe chronic lung disease, or dialysis) were adjudicated by surgeons and data managers from all 14 programs, as previously described for Mass-DAC percutaneous coronary intervention (PCI) public reporting.
      • Barringhaus K.G.
      • Zelevinsky K.
      • Lovett A.
      • Normand S.L.
      • Ho K.K.
      Impact of independent data adjudication on hospital-specific estimates of risk-adjusted mortality following percutaneous coronary interventions in massachusetts.
      Using published procedure classification criteria
      Massachusett Department of Health
      Cardiac services and treatments.
      Massachusetts Data Analysis Center
      Adult coronary artery bypass graft surgery FY14 annual report.
      and clinical review of medical records and operative notes, all procedures coded “CABG + other” were also adjudicated, based on the magnitude and incremental risk of the “other” procedure, to confirm they were not isolated CABG cases that were intentionally or inadvertently coded as combined procedures (that are not publicly reported).

      Expected Mortality and Outlier Determination

      STS cohort

      For operational feedback reports, STS recalibrates its risk models to achieve a ratio of observed to expected mortality (O/E) of 1 and periodically estimates completely new models.
      • Shahian D.M.
      • O'Brien S.M.
      • Filardo G.
      • Ferraris V.A.
      • Haan C.K.
      • Rich J.B.
      • et al.
      The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery.
      • O'Brien S.M.
      • Shahian D.M.
      • Filardo G.
      • Ferraris V.A.
      • Haan C.K.
      • Rich J.B.
      • et al.
      The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery.
      • Shahian D.M.
      • O'Brien S.M.
      • Filardo G.
      • Ferraris V.A.
      • Haan C.K.
      • Rich J.B.
      • et al.
      The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3—valve plus coronary artery bypass grafting surgery.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • Kurlansky P.A.
      • Furnary A.P.
      • Cleveland Jr., J.C.
      • et al.
      The Society of Thoracic Surgeons 2018 adult cardiac surgery risk models: part 1-background, design considerations, and model development.
      • O'Brien S.M.
      • Feng L.
      • He X.
      • Xian Y.
      • Jacobs J.P.
      • Badhwar V.
      • et al.
      The Society of Thoracic Surgeons 2018 adult cardiac surgery risk models: part 2-statistical methods and results.
      Depending on the specific procedure and outcome measure, outlier determination is typically based on 95% or 98% Bayesian credible intervals.
      • O'Brien S.M.
      • Shahian D.M.
      • DeLong E.R.
      • Normand S.L.
      • Edwards F.H.
      • Ferraris V.A.
      • et al.
      Quality measurement in adult cardiac surgery: part 2—statistical considerations in composite measure scoring and provider rating.
      For this study, to facilitate longitudinal analyses of expected mortality trends, the same 2008 STS CABG risk model
      • Shahian D.M.
      • O'Brien S.M.
      • Filardo G.
      • Ferraris V.A.
      • Haan C.K.
      • Rich J.B.
      • et al.
      The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery.
      was applied to patients from each year.

      Mass-DAC cohort

      Mass-DAC develops its own 30-day mortality risk models based solely on data from Massachusetts programs and re-estimates these annually.
      Massachusett Department of Health
      Cardiac services and treatments.
      Massachusetts Data Analysis Center
      Adult coronary artery bypass graft surgery FY14 annual report.
      Because of frequent re-estimation and only 14 programs, annual Massachusetts O/E ratios are typically close to unity.
      Fully Bayesian hierarchical logistic (surgery years 2002-2009) and Poisson (surgical procedures beginning in 2010) regression models are estimated that include hospital-specific random intercepts.
      Hospitals were categorized as outliers if their standardized mortality incidence rate 95% credible interval excluded the statewide average, or if their observed mortality rate was statistically different from that predicted by all other hospitals (ie, with the problematic hospital's data excluded from risk model development).
      The Massachusetts Cardiac Care Outlier Committee (composed of surgeons from the Massachusetts STS) reviewed Mass-DAC annual results before final release, including clinical information from statistical outliers, to determine whether there were systemic versus unique case mix issues to explain low performance.

      Statistical Analyses

      We compared longitudinal trends in Massachusetts isolated CABG volume with trends from STS (national) and from New York, another public reporting state.
      Expected and adjusted mortality rate analyses comparing Massachusetts and national [STS] cohorts used MA [STS] data (similar to Mass-DAC data) and STS predicted risk of operative mortality (PROM). Risk factor prevalence comparisons primarily used peer-adjudicated Mass-DAC data, supplemented by MA [STS] data.
      We analyzed annual observed operative and 30-day mortality rates for all 3 study cohorts; expected operative mortality rates and O/E ratios for national [STS] and MA [STS] cohorts, using STS risk models; and expected 30-day mortality for the Mass-DAC cohort.
      To statistically compare national [STS] and Massachusetts risk factor prevalences, surgical outcomes, and their corresponding longitudinal changes, we estimated separate logistic regression models for 15 selected risk factors and for observed operative and 30-day mortality. We linked the log-odds of each end point (ie, risk factor or mortality) to categorical variables indicating Massachusetts (Mass-DAC data source for risk factors, MA [STS] data source for mortality) versus national [STS] data, year of surgery (year 0 [2003] to year 11 [2014]), and an interaction of Massachusetts indicator and year; for mortality, these regressions were also estimated with adjustment for STS PROM, a summary risk score based on patient factors. Significant odds ratios (ORs) for interaction terms imply that annual rates of change in mortality or risk factor odds differed between Massachusetts and national [STS] data; when interaction terms are nonsignificant, the MA indicator represents the overall OR between Massachusetts and national [STS], and the year indicator is the overall rate of change. We noted if annual trends were increasing or decreasing in both cohorts, or directionally opposite. To minimize family-wise error rate, we used conservative 99% confidence intervals.
      We explored hospital-specific percentage shares of Massachusetts CABG volumes between 2003 and 2014, changes in the share of CABG cases at 2 outlier hospitals, and expected mortality rate trends of outlier hospitals compared with the volume weighted average expected mortality (Mass-DAC models) for all Massachusetts hospitals.
      Mass-DAC used hospital-aggregated, de-identified data. Raw data and procedures associated with assessing hospital quality were reviewed by the Harvard Medical School Institutional Review Board. STS data were likewise de-identified, and STS data collection and analysis procedures were reviewed by the Duke Institutional Review Board.

      Results

      CABG Volumes

      Massachusetts isolated CABG admissions declined 33.5% from 2002 (n = 4603) to 2014 (n = 3063), less than the 56.1% decrease in comparable STS procedures/program nationally, the 52.7% decrease among 388 sites participating continuously in the STS ACSD from 2002 to 2014, and the 50.7% decrease in New York (Table E2).

      Mass-DAC Verification of Case Completeness and Deaths

      Between 2004 and 2014, Mass-DAC verification of case completeness and vital status resulted in 0 to 2 (mean, 0.9) additional isolated CABG procedures (0.03% of all cases) and 0 to 5 (mean, 2) net 30-day mortalities (0.06% of all cases) added annually to data submitted by hospitals.

      Mass-DAC Adjudication

      From 2002 to 2014, 541 to 781 total CABG cases (mean, 636.8) were reviewed annually for various indications, including 273 to 724 risk variable (mean, 445.8) audits. Excluding 2002, 38-138 (mean, 82.7) risk variables annually had coding changes.
      Detailed annual adjudication data are available from 2007 to 2014 (Table E3) for high-risk variables and to substantiate CABG + other procedure designation. Compared with the 9.8% decline in isolated CABG volume during this period, cases submitted with shock or emergent/emergent-salvage status declined substantially more (eg, shock, 67% decrease; emergent/emergent-salvage, 41% decrease), as did their submission rates as percentages of isolated CABG cases and their rates of coding changes.
      Between 2007 and 2014, 70 to 174 (mean, 122) procedures annually were coded by hospitals as CABG + other. Both the number (149 to 98) of submitted CABG + other cases and the number changed to isolated CABG (93 to 55) decreased between 2007-2014; overall, between 33.1% and 65.4% CABG + other cases were reclassified isolated CABG.

      Mass-DAC Versus STS Risk Factor Prevalence

      Massachusetts risk factor prevalences (ie, Mass-DAC and MA [STS]) are compared with national [STS] prevalences between 2003 and 2014 in Table 1 and Figure E1, Figure E2, Figure E3, Figure E4, Figure E5. Mass-DAC and MA [STS] rates closely agreed for most, but not all (eg, several peer-adjudicated variables) risk factors.
      Table 1Risk factor prevalences, Massachusetts versus national [STS]
      YearMean age (y)Age 65- 74 y (%)Age ≥75 y (%)Female (%)EF 0.30- 0.39 (%)EF < 0.30 (%)CVD (%)
      STSMass-DACSTSMass-DACSTSMass-DACSTSMass-DACSTSMass-DACSTSMass-DACSTSMass-DAC
      200365.066.731.632.121.826.827.926.510.712.46.47.713.514.0
      200465.166.931.532.222.227.827.925.510.411.06.38.414.014.8
      200565.166.431.331.222.026.027.223.610.510.96.48.113.714.1
      200665.066.331.531.321.525.627.324.410.19.96.36.313.713.7
      200764.966.031.330.721.025.127.124.310.110.96.36.813.714.0
      200864.866.031.831.820.424.527.322.49.610.16.25.514.212.9
      200964.866.132.433.020.023.826.922.59.49.16.26.414.314.8
      201064.865.832.732.119.923.026.322.99.39.46.06.214.513.3
      201164.966.232.932.419.723.926.422.49.39.96.15.814.115.4
      201264.866.133.835.119.222.126.122.29.59.16.25.514.114.2
      201365.066.334.632.719.424.525.722.49.48.86.35.814.213.9
      201464.966.534.636.318.923.525.122.09.48.36.45.916.517.2
      Dialysis (%)Morbid obesity (%)Reoperation (%)Moderate/severe CLD (%)Diabetes (%)Insulin dependent (%)
      STSMass-DACSTSMass-DACSTSMass-DACSTSMass-DACSTSMass-DACSTSMass-DAC
      20031.51.314.312.09.06.035.838.110.212.2
      20041.61.414.312.05.32.59.25.636.237.010.510.6
      20051.71.214.912.84.82.69.55.536.839.311.211.7
      20061.81.815.313.74.62.29.55.937.640.011.412.5
      20071.91.915.814.34.42.69.65.839.141.812.114.4
      20082.42.116.313.44.12.79.96.339.737.312.911.8
      20092.52.416.715.03.91.710.15.140.539.313.612.9
      20102.62.316.815.23.81.810.45.841.439.714.114.6
      20112.82.017.116.13.62.110.45.843.141.515.015.1
      20122.81.617.415.83.41.310.74.844.943.115.915.9
      20132.82.217.316.23.11.410.74.745.744.516.917.4
      20143.02.317.715.62.91.410.15.647.343.817.517.6
      Cardiogenic shock (%)Emergent ± salvage (%)MI <24 h (%)PVD (%)
      STSMass-DACMA [STS]STSMass-DACMA [STS]STSMass-DACMA [STS]STSMass-DACMA [STS]
      20031.91.62.14.63.03.83.93.13.115.717.417.4
      20041.91.12.24.83.14.54.23.84.215.417.717.8
      20052.01.01.95.02.53.44.52.93.615.517.517.3
      20062.00.81.75.02.73.54.63.74.215.317.717.7
      20072.00.91.64.93.44.24.73.14.115.117.117.1
      20082.00.71.44.73.03.34.43.33.714.814.114.0
      20091.90.61.14.62.42.64.32.42.715.015.315.2
      20102.00.51.04.52.52.84.32.62.815.115.515.5
      20112.00.40.64.72.62.94.22.72.814.615.515.8
      20122.00.70.65.02.52.64.02.42.314.211.814.1
      20131.80.80.94.82.63.23.92.82.414.112.614.1
      20142.00.60.84.82.52.63.62.52.214.412.514.5
      For 4 risk factors that underwent peer-adjudication in Massachusetts, Table 1 includes prevalences of those factors in data submitted directly from Massachusetts programs to the Society of Thoracic Surgeons (STS) national database (MA [STS]), as well as the comparable prevalences in Massachusetts Data Analysis Center (Mass-DAC) data following peer-adjudication. EF, Ejection fraction; CVD, cerebrovascular disease; STS, Society of Thoracic Surgeons; DAC, Data Analysis Center; CLD, chronic lung disease; MI, myocardial infarction; PVD, peripheral vascular disease.
      Regression analysis results of selected risk factors are summarized in Figure 1 and Table E4. Relative to national [STS] patients, adjudicated Mass-DAC data showed significantly lower odds of cardiogenic shock, reoperation, emergency/emergency-salvage, chronic lung disease, dialysis, morbid obesity, and female sex; and significantly greater odds of diabetes, PVD, low EF, age ≥75 years (Figure 1, B). Annual rates of change in risk factor odds in Massachusetts relative to national [STS] data were not statistically different for 8 of 15 risk factors studied. Odds of cardiogenic shock decreased significantly in Massachusetts relative to national [STS] rates, which were slightly increasing during the study period [Figure 1, C]). Odds of diabetes increased at a slightly slower rate in Massachusetts than nationally, whereas odds of PVD, EF 0.30 to 0.39 and EF <0.30, and female sex fell faster in Massachusetts.
      Figure thumbnail gr1
      Figure 1Multivariable logistic regression analysis of selected risk factor odds, 2003 to 2014. To statistically compare national [STS] and Massachusetts risk factor prevalences and their longitudinal changes, we estimated separate logistic regression models for 15 selected risk factors. We linked the log-odds of each risk factor to categorical variables indicating Massachusetts Data Analysis Center (Mass-DAC) versus national STS data (excluding MA patients), year of surgery (year 0 [2003] to year 11 [2014]), and an interaction of Mass-DAC indicator and year. Significant odds ratios for interaction terms imply that annual rates of change in risk factor odds differed between Massachusetts and national [STS] data; when interaction terms are non-significant, the MA indicator represents the overall odds ratio between Massachusetts and national [STS], and the year indicator is the overall rate of change. We noted if annual trends were increasing or decreasing in both cohorts, or directionally opposite. Odds ratios with 99% confidence intervals (CI) are summarized for the 3 model co-variables: A, Massachusetts (MA) (Yes/No) = MA relative to national STS. B, Year = Annual change (2003 = 0 increasing to 2014 = 11). C, MA × Year (interaction term) = Annual change in MA relative to annual change in national STS. STS, Society of Thoracic Surgeons; Mod/Sev CLD, Moderate or severe chronic lung disease; PVD, peripheral vascular disease; CVD, cerebrovascular disease; EF, ejection fraction; Pre-op, preoperative.

      Operative and 30-Day Mortality

      Massachusetts observed operative (Figure 2, A) and 30-day (Figure E6) mortality rates were no different than national [STS] rates in 2003, then became significantly and consistently lower than STS national rates, although they increased after 2011. For several annual reports of operative and 30-day mortality, Mass-DAC observed mortality rates were slightly higher than MA [STS] rates, probably reflecting enhanced vital status verification.
      Figure thumbnail gr2
      Figure 2Expected and observed mortality rates. A, Observed operative mortality for the Society of Thoracic Surgeons (STS) national cohort (excluding Massachusetts (MA) patients), MA patient data submitted by hospitals directly to STS (MA [STS]), and Massachusetts data adjudicated and verified by the Massachusetts Data Analysis Center (Mass-DAC), which captured a few more deaths in certain years. B, Expected operative mortality rates (in-hospital + 30-day) for the national [STS] cohort and MA [STS] cohorts, based on STS data and STS 2008 risk models.
      Using the STS isolated CABG risk model, MA [STS] PROM rates were higher than national [STS] rates from 2003 to 2007 (P < .001) but declining, compared with relatively flat national [STS] expected rates; from 2008-2014, MA [STS] and national [STS] PROM rates were similar (P = .135) and stable (Figure 2, B). Using STS data and risk models, MA [STS] O/E for operative mortality generally ranged from 0.6 to 0.8 (Figure E7).
      Because Mass-DAC risk models for Massachusetts public reporting were derived only from Massachusetts programs and re-estimated annually, observed and expected 30-day mortality rates tracked very closely (Figure E8).
      In regression analyses incorporating STS PROM, the MA [STS] CABG cohort had significantly lower adjusted odds of operative (OR, 0.82; 99% confidence interval, 0.69-0.98) and 30-day (OR, 0.79; 99% confidence interval, 0.66-0.96) mortality relative to the national [STS] cohort (Figure 3), with no significant difference in the annual change in MA [STS] mortality odds relative to the annual change in national [STS] cohort odds (MA × Year interaction term).
      Figure thumbnail gr3
      Figure 3Operative and 30-day mortality odds and relative change in odds, Massachusetts STS (MA [STS]) data versus national STS data (excluding MA patients). To statistically compare national and Massachusetts surgical outcomes and their longitudinal changes, we estimated logistic regression models for observed operative and 30-day mortality. We linked the log-odds of each end point to categorical variables indicating MA [STS] versus national [STS] data, year of surgery (Year: 0 [2003] to 11 [2014]), and an interaction of MA indicator and year. These regressions were also estimated with adjustment for STS predicted risk of mortality, a summary risk score based on patient factors. Significant odds ratios for interaction terms imply that annual rates of change in mortality differed between Massachusetts and national data; when interaction terms are nonsignificant, the MA indicator represents the overall odds ratio between Massachusetts and national data, and the year indicator is the overall rate of change. Multivariable logistic regression analysis results shown for operative (red) and 30-day (blue) mortality outcomes without (unadjusted, open symbols) and with (adjusted, closed symbols) risk-adjustment using 2008 STS predicted risk of operative mortality model. Covariates: MA (Yes/No) = MA relative to national STS; Year = annual change (2003 = 0 increasing to 2014 = 11); MA × Year (interaction term) = annual change in MA mortality odds relative to annual change in national STS mortality odds. OR, Odds ratio; 99% CI, 99% confidence interval.

      Surgeon-Level Reports

      Massachusetts provided surgeon-level reports for procedures performed between 2002 and 2010 (last report February 2012 based on fiscal year [FY] 2008-2010 data). Two high-mortality surgeon outliers were identified in calendar year (CY) 2002-2004 report (published 2006), 3 in the CY2003-CY2005 report (published in 2007), and 2 in the FY2003-FY2006 report (published in February 2008). Because of overlapping 3-year time windows, the same outliers were identified in several reports. There were only 3 unique outlier surgeons, all at outlier hospitals.

      Consequences of Outlier Designation

      Subsequent expected mortality rates

      Two Massachusetts hospitals were transiently designated as 30-day high mortality outliers—Hospital A in 2005 and 2007 reports (2003 and 2005 procedures); and Hospital B in the 2006 report (2004 procedures). Before outlier designation, both hospitals had higher expected mortality rates than the volume weighted state average (Figure 4).
      Figure thumbnail gr4
      Figure 4Outlier designation and expected mortality rates. Before outlier designation, both hospitals had higher expected mortality rates than the volume weighted state average. As a result of its outlier designation, Hospital A temporarily suspended its cardiac program and made extensive, state-recommended changes. Although again an outlier 2 years later, their results were substantially improving. Hospital A's expected mortality decreased after its initial outlier designation; by 2010, its expected mortality rate had returned to the state average, which it tracked closely thereafter. Hospital B's expected mortality increased slightly following outlier designation, then converged towards and tracked the state mean from 2010-2011 onward.
      Massachusetts DPH temporarily closed Hospital A's program after its first outlier designation, leading to extensive structural, personnel, and leadership changes. Although again an outlier 2 years later, their results were substantially improving. Hospital A's expected mortality decreased after its initial outlier designation, but by 2010 its expected mortality rate had returned to the state average, which it tracked closely thereafter (Figure 4).
      Hospital B's expected mortality increased slightly following outlier designation, then converged toward and tracked the state mean from 2010 to 2011 onward.

      Share of Massachusetts CABG Cases

      Following initial outlier designation (CY2003 report, published in October 2005) and temporary closure, Hospital A's share of all Massachusetts CABG cases dropped from 6.5% (2005) to 4.9% (2006). With implementation of numerous changes recommended by a state-mandated panel of experts, outcomes improved and their percentage share of Massachusetts CABG cases rebounded to 7.1% in 2007, with steady increases thereafter despite their second identification as an outlier (October 2007, based on CY2005 outcomes) (Figure 5).
      Figure thumbnail gr5
      Figure 5Outlier designation and percentage share of total Massachusetts coronary artery bypass graft (CABG) cases. Following initial outlier designation and temporary closure, Hospital A's share of Massachusetts CABG cases decreased. After implementation of numerous changes, their outcomes improved and their percentage share of Massachusetts CABG cases rebounded in 2007, steadily increasing thereafter despite their second identification as an outlier. Hospital B experienced growth in its share of Massachusetts CABG volume the year following its outlier designation; this percentage then stabilized for 4 years and subsequently increased.
      Hospital B experienced growth in its share of Massachusetts CABG volume from 4.4% to 4.7% the year following its outlier designation (CY2004 report, published October 2006); this percentage then stabilized for 4 years and subsequently increased.
      Overall, both Hospitals A and B, despite their transient outlier status, experienced substantial increases in their percentage shares of Massachusetts CABG volume from 2003 to 2014 (18.7% and 37.1%, respectively) (Figure 6).
      Figure thumbnail gr6
      Figure 6Hospital-specific percentage share of total Massachusetts coronary artery bypass graft (CABG) cases, 2003 versus 2014. Between 2003 and 2014, 8 of 14 Massachusetts adult cardiac surgery programs experienced volume growth, including both programs that had been transient outliers. Relative volume shifts among hospitals were more related to initiation of 3 new cardiac surgery programs, which drew volume from existing programs, and with altered referral patterns at several hospitals due to leadership and staff changes.

      Discussion

      Massachusetts observed mortality was significantly and consistently less than the STS PROM for its case-mix (O/E < 1), and adjusted mortality odds were about 20% lower than national STS results. These findings could be associated with public reporting, especially since Massachusetts and STS national rates were similar at the beginning of the public reporting experience (Figure 2, A). However, a direct causal association cannot be inferred from these data, and there are numerous other confounders. For example, Massachusetts also has a longstanding Determination of Need (referred to in other states as Certificate of Need) program, which some states use to restrict the number of hospitals offering certain high cost services
      • Ho V.
      • Ku-Goto M.H.
      • Jollis J.G.
      Certificate of Need (CON) for cardiac care: controversy over the contributions of CON.
      • DiSesa V.J.
      • O'Brien S.M.
      • Welke K.F.
      • Beland S.M.
      • Haan C.K.
      • Vaughan-Sarrazin M.S.
      • et al.
      Contemporary impact of state certificate-of-need regulations for cardiac surgery: an analysis using the Society of Thoracic Surgeons' National Cardiac Surgery Database.
      ; in Massachusetts, this resulted in relatively fewer but higher volume cardiac surgery programs. Furthermore, most Massachusetts cardiac surgery units are based in major academic centers or their affiliates. Both higher volume and teaching intensity have been associated with improved health care outcomes.
      • Birkmeyer J.D.
      • Stukel T.A.
      • Siewers A.E.
      • Goodney P.P.
      • Wennberg D.E.
      • Lucas F.L.
      Surgeon volume and operative mortality in the United States.
      • Birkmeyer J.D.
      • Siewers A.E.
      • Finlayson E.V.A.
      • Stukel T.A.
      • Lucas F.L.
      • Batista I.
      • et al.
      Hospital volume and surgical mortality in the United States.
      • Shahian D.M.
      • Normand S.L.
      The volume-outcome relationship: from Luft to Leapfrog.
      • Burke L.G.
      • Frakt A.B.
      • Khullar D.
      • Orav E.J.
      • Jha A.K.
      Association between teaching status and mortality in US Hospitals.
      • Shahian D.M.
      • Liu X.
      • Meyer G.S.
      • Torchiana D.F.
      • Normand S.L.
      Hospital teaching intensity and mortality for acute myocardial infarction, heart failure, and pneumonia.
      Risk aversion is the most frequently discussed and troubling unintended consequence of public reporting,
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. part 1: observations from cardiac surgery and interventional cardiology.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. Part 2: mitigation strategies.
      and this is another potential explanation for lower observed mortality rates in Massachusetts. Risk aversion is typically described as the inappropriate denial of care to high-risk patients who may benefit most from interventions. Not fully trusting the adequacy of risk-adjustment, some providers may be concerned that the anticipated worse outcomes of very high-risk patients will be reflected in their report card scores, thereby harming their practices and reputations. Although the existence and extent of this phenomenon was vigorously debated in the early New York and Pennsylvania CABG public reporting experience, the overall evidence for risk aversion in cardiac surgery is inconclusive, although it is well documented in PCI for complicated acute myocardial infarction.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. part 1: observations from cardiac surgery and interventional cardiology.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. Part 2: mitigation strategies.
      • Waldo S.W.
      • McCabe J.M.
      • O'Brien C.
      • Kennedy K.F.
      • Joynt K.E.
      • Yeh R.W.
      Association between public reporting of outcomes with procedural management and mortality for patients with acute myocardial infarction.
      • McCabe J.M.
      • Waldo S.W.
      • Kennedy K.F.
      • Yeh R.W.
      Treatment and outcomes of acute myocardial infarction complicated by shock after public reporting policy changes in New York.
      • Joynt K.E.
      • Blumenthal D.M.
      • Orav E.J.
      • Resnic F.S.
      • Jha A.K.
      Association of public reporting for percutaneous coronary intervention with utilization and outcomes among Medicare beneficiaries with acute myocardial infarction.
      In the current study, the lower odds of several important risk factors in Massachusetts (eg, shock, emergent/emergent-salvage, and reoperation) could reflect risk aversion as a potential explanation for the lower observed CABG mortality in Massachusetts compared with the rest of the United States. However, peer-adjudication of risk factors in Massachusetts likely contributed in part to these lower prevalences, both by educating surgeons and data managers regarding definitional nuances, and when necessary by downcoding of submitted data. Other more common risk factors (eg, advanced age, low EF, diabetes, and PVD) had higher or similar prevalences in Massachusetts; temporal trends for many risk factors were no different.
      When compared with relatively stable national [STS] rates, decreasing Massachusetts expected mortality from 2003 to 2007 could also be interpreted as supporting risk aversion. However, using identical STS risk models, MA [STS] expected mortality rates were significantly higher than national [STS] rates for this period, although declining, and similar from 2008 onward, which argues against risk aversion. The timeline of Massachusetts public reporting provides useful context for these observations. Massachusetts reporting legislation was passed in 2000 and immediately energized the cardiac surgery community. Two years of anticipation preceded the first year (2002) in which data were actually collected and analyzed. In this context, the significantly higher expected mortality rates for Massachusetts (MA [STS]) compared with national [STS] during the early years of public reporting, when there was maximum hospital and surgeon anxiety) seems inconsistent with substantial risk averse behavior.
      The increase in MA [STS] observed mortality after 2011 could simply represent regression to the mean after exceptionally low 2011 observed rates. However, there was also a temporal association with phasing out of surgeon-level reporting during the preceding years, which some would argue made surgeons less reluctant to accept very high-risk patients.
      Overall, our empirical findings are conflicting and inconclusive regarding risk aversion, at least at the aggregate level. Available Massachusetts data provide no insights regarding individual patient-level decisions, although anecdotal evidence suggests this phenomenon does sometimes occur, even if not at a detectable frequency within the CABG population.
      Surgeon and data manager involvement with and support of the Massachusetts public reporting program were critical to its success. For example, Massachusetts’ peer adjudication provided additional coding review beyond standard STS audit processes and could be a model for other states and regions. Over time, progressive decreases in cases submitted with selected high-risk preoperative variables, and fewer changes in these adjudicated variables and in cases coded CABG + other, suggest the educational value of this exercise. The greatest divergence between adjudicated Mass-DAC risk factor coding and data submitted directly to STS occurred in the early years of Massachusetts public reporting. This probably reflects the initial coding inexperience of Massachusetts data managers and surgeons (most of whom had not previously participated in the STS Database) as well as some less precise STS variable definitions in earlier years. The Massachusetts peer adjudication process educated all stakeholders regarding optimal coding practices and nuances. It also made them aware that their coding would be adjudicated by peers, making intentional upcoding less likely. Contemporaneously, national STS resources to assist data managers with coding issues were expanded, and STS external audit processes became more robust. In the 2016 external national audit report of the STS ACSD, auditors agreed with submitted coding of cardiogenic shock, myocardial infarction timing, and status (ie, elective, urgent, emergent, or emergent/salvage) in 99.68%, 97.59%, and 97.63% of charts audited, respectively. Thus, because of both Massachusetts and national initiatives, coding for high-risk variables in these 2 data sources progressively converged and became more accurate, although there is a persistently lower prevalence of several such variables in Massachusetts.
      Mass-DAC case completeness and mortality verification confirm that hospitals were not hiding cases or deaths; they demonstrate the utility of linking clinical registry data to state and national administrative sources for external validation.
      The tendency for outliers to become risk averse, previously documented for PCI,
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. part 1: observations from cardiac surgery and interventional cardiology.
      • Shahian D.M.
      • Jacobs J.P.
      • Badhwar V.
      • D'Agostino R.S.
      • Bavaria J.E.
      • Prager R.L.
      Risk aversion and public reporting. Part 2: mitigation strategies.
      • Waldo S.W.
      • McCabe J.M.
      • O'Brien C.
      • Kennedy K.F.
      • Joynt K.E.
      • Yeh R.W.
      Association between public reporting of outcomes with procedural management and mortality for patients with acute myocardial infarction.
      • McCabe J.M.
      • Joynt K.E.
      • Welt F.G.
      • Resnic F.S.
      Impact of public reporting and outlier status identification on percutaneous coronary intervention case selection in Massachusetts.
      was transient and inconsistent in our study. Within a few years after outlier designation, expected mortality rates at both Massachusetts outlier programs had stabilized around the state average.
      Similar to New York findings,
      • Chassin M.R.
      Achieving and sustaining improved quality: lessons from New York State and cardiac surgery.
      • Jha A.K.
      • Epstein A.M.
      The predictive accuracy of the New York State coronary artery bypass surgery report-card system.
      • Mukamel D.B.
      • Weimer D.L.
      • Mushlin A.I.
      Interpreting market share changes as evidence for effectiveness of quality report cards.
      • Mukamel D.B.
      • Mushlin A.I.
      The impact of quality report cards on choice of physicians, hospitals, and HMOs: a midcourse evaluation.
      • Mukamel D.B.
      • Mushlin A.I.
      Quality of care information makes a difference: an analysis of market share and price changes after publication of the New York State Cardiac Surgery Mortality Reports.
      only 1 of the 2 outlier hospitals experienced a negative influence on its share of Massachusetts CABG cases and that effect was transient. Many factors besides objective outcomes influence patient selection of a provider, including the recommendations of the referring physicians, family, and friends; geographic preferences; differences in copays; and limited choices available in their insurance plan. Relative volume shifts among hospitals were more related to initiation of 3 new cardiac surgery programs, which drew volume from existing programs, and with altered referral patterns at several hospitals due to leadership and staff changes. If underlying causes of outlier designation are addressed quickly and effectively, referral impact should be limited and transient.

      Limitations

      One major motivation for the Commonwealth of Massachusetts to mandate participation in the STS National Database was the prior lack of credible data regarding cardiac surgical outcomes. Thus, we have no data on risk factors or outcomes before the institution of the public reporting program, which limits our ability to comment upon comparisons of Massachusetts and national data before 2002.
      During the study period, STS practice was to regard postdischarge patients with unknown or missing 30-day vital status as alive, which could potentially introduce bias. However, in-hospital deaths are recorded with near 100% completeness and these represent the vast majority of 30-day deaths. STS simulations (available upon request) suggest that any errors introduced by the false assumption that a patient with missing or unknown 30-day status is alive have negligible influence on hospital mortality results compared with random sampling error. Also, given 100% complete 30-day vital status for Massachusetts patients, if any STS national patient were misclassified as alive, correcting this would only increase the apparent mortality advantage for Massachusetts.
      We do not have data on patients who may have not been offered surgery because of their high risk, which limits our assessment of risk aversion.

      Conclusions

      Mandatory Massachusetts public reporting did not disproportionately limit overall access to cardiac surgery services, as demonstrated by overall CABG volume trends from 2002 to 2014. Moreover, using comparable data and risk models, CABG outcomes in Massachusetts from 2003 to 2014 were consistently and significantly superior to national [STS] results. However, because extensive longitudinal data are unavailable before the initiation of Massachusetts public reporting, we cannot assume a causal association of report cards and superior outcomes. Finally, evidence for risk aversion was inconsistent and inconclusive. These findings, together with previous studies from New York and from STS, support the continued study of public reporting as 1 mechanism to assist patients in selecting providers and to promote optimal surgical outcomes.
      Our experience suggests that surgeons in public reporting states may benefit from the establishment of formal peer organizations (such as the MA STS) that can serve as their interface with state regulatory agencies. This allows them to be part of the measure development process, to provide clinical context for problematic statistical results, and to offer remedial recommendations for outlier programs. Robust surgeon involvement in the public reporting process was also evident in the Massachusetts peer adjudication process for risk factor coding. This provides valuable oversight and education regarding appropriate coding practices, reassures surgeons and hospitals regarding data integrity, and may mitigate risk aversion by fostering provider trust in the system. It is most easily operationalized at the state or regional levels.
      Finally, the Massachusetts experience demonstrates that if outlier hospitals implement appropriate programmatic, structural, leadership, and staff changes, the negative consequences of their outlier status may be temporary and remediable.

      Webcast

      Conflict of Interest Statement

      Drs Shahian, Torchiana, D'Agostino, Engelman, Rawn, and Birjiniuk have been unpaid, volunteer leaders in the development and implementation of the Massachusetts public reporting system. Dr Normand has been a paid leader for and Ms Lovett and Mr Cioffi are employees of Mass-DAC, the entity that administered the Massachusetts cardiac surgery public reporting program during the term of this study. Dr Habib is director of the Society of Thoracic Surgeons Research Center. All other authors have nothing to disclose with regard to commercial support.
      The authors thank the Massachusetts Department of Public Health for the use of their data, and the Society of Thoracic Surgeons for providing data from its Adult Cardiac Surgery Database.

      Appendix

      Figure thumbnail fx3
      Figure E1Risk prevalence plots for age and ejection fraction. For these 4 risk factors, MA [STS] (MA data submitted directly to STS) and Mass-DAC prevalence rates were virtually superimposable. These rates were generally higher than national STS rates (excluding MA patients). MA and national STS temporal trends in these risk factors were similar except for a sharp decrease in EF < 0.30 prevalences in MA patients from 2003-2006. STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
      Figure thumbnail fx4
      Figure E2Risk prevalence plots for age and comorbid conditions. For these 4 risk factors, MA [STS] (MA data submitted directly to STS) and Mass-DAC prevalence rates were nearly identical. These rates and their temporal trends were generally similar to those of national STS data (excluding MA patients). STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
      Figure thumbnail fx5
      Figure E3Risk prevalence plots for sex and comorbid conditions. For these 4 risk factors, MA [STS] (data submitted directly to STS) and Mass-DAC prevalence data were nearly identical. Massachusetts prevalence rates were consistently less than national STS rates (excluding MA). Temporal trends were similar between Massachusetts and national data for female sex, reoperation, and morbid obesity, but divergent for chronic lung disease (national STS slightly increasing, Massachusetts stable to slightly decreasing). STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
      Figure thumbnail fx6
      Figure E4Episodic peer adjudication. Peripheral vascular disease and preoperative dialysis underwent additional peer-adjudication only for a few years (circled in the plots). During the period of adjudication, peripheral vascular disease prevalences were lower in Mass-DAC data (additional peer adjudication) than in unadjudicated MA [STS] data or national [STS] data (excluding MA). Preoperative dialysis prevalence rates were virtually identical in adjudicated Mass-DAC data and MA data submitted directly to STS (MA[STS]), and these rates were lower than national [STS] prevalence rates. STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
      Figure thumbnail fx7
      Figure E5Routinely peer-adjudicated variables. For all three routinely peer-adjudicated risk factors, Massachusetts Data Analysis Center (Mass-DAC) prevalence rates were lower than MA [STS] rates (submitted directly to STS) in the early years of MA public reporting. With increasing experience and education of data managers and surgeons, the Mass-DAC and MA[STS] rates converged (vertical dotted line) for all 3 risk factors but remained substantially lower than national STS rates (excluding MA). MI, Myocardial infarction; STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
      Figure thumbnail fx8
      Figure E6Massachusetts and national Society of Thoracic Surgeons (STS) 30-day observed mortality rates, 2003 to 2014. In 2003, observed 30-day mortality rates were similar for MA CABG patients (whether data were submitted directly to STS (MA [STS]) or analyzed by Mass-DAC) compared with national STS mortality rates (excluding MA). MA rates then decreased substantially and were consistently lower than national STS rates thereafter, although they increased after 2011. MA [STS], Massachusetts data submitted directly to STS; Mass-DAC, Massachusetts Data Analysis Center.
      Figure thumbnail fx9
      Figure E7Ratios of observed to expected operative mortality for CABG, Massachusetts versus national STS results. For each public reporting year, using the same STS 2008 risk models, the ratios of observed to expected (O/E) operative mortality for MA patients in the STS Database were consistently less than one and also lower than national STS O/E ratios (excluding MA patients).
      Figure thumbnail fx10
      Figure E8Massachusetts Data Analysis Center (Mass-DAC) 30-day observed and expected CABG mortality rates. Because Mass-DAC expected mortality rates were derived from risk models based only on the 14 MA programs and were re-estimated annually, observed and expected rates tracked closely.
      Table E1Vital status ascertainment or verification
      Table E2Isolated coronary artery bypass grafting (CABG) volumes: Society of Thoracic Surgeons (STS) national (without Massachusetts [MA]) per program and STS national (without MA) for 388 continuously participating STS programs, 2002 to 2014; New York decrease 2002 to 2014
      YearSTS national data (overall)
      Data from all sites contributing to STS Adult Cardiac Surgery Database other than Massachusetts (out of 1253 possible sites).
      STS national data (continuous participation)
      Data from 388 STS Adult Cardiac Surgery Database sites that reported cases for all years between 2002 and 2014. New York CABG volumesNew York: 16,120 isolated CABG in 2002, 7942 in 2014 = 50.7% decrease. New York Data Sources from: 1. 2004 Adult cardiac surgery in New York State 2000-2002. Available at: https://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2000-2002_adult_cardiac_surgery.pdf. Accessed September 29, 2017. 2. 2017 Adult cardiac surgery in New York State 2012-2014. Available at: https://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2012-2014_adult_cardiac_surgery.pdf. Accessed September 29, 2017.
      No. of sitesIsolated CABG (n)No. of sitesIsolated CABG (n)
      2002495151,799388124,037
      2003539150,801388116,379
      2004607147,884388105,409
      2005704150,61638894,890
      2006787158,04638889,513
      2007859161,16738884,549
      2008933165,05538880,115
      2009965164,22838875,593
      2010996157,83838871,251
      20111016144,71838864,189
      20121026138,40838860,094
      20131038138,77138860,146
      20141041137,70238859,104
      20151045141,03938859,369
      20161046140,85338858,672
      Decrease per site STS overall = (151,799/495-140,853/1046)/306.7 = 56.1%. Decrease at 388 sites with continuous STS participation 2002 to 2014 = (124,037-58,672)/124,037 = 52.7%. CABG, Coronary artery bypass grafting.
      Data from all sites contributing to STS Adult Cardiac Surgery Database other than Massachusetts (out of 1253 possible sites).
      Data from 388 STS Adult Cardiac Surgery Database sites that reported cases for all years between 2002 and 2014. New York CABG volumesNew York: 16,120 isolated CABG in 2002, 7942 in 2014 = 50.7% decrease. New York Data Sources from: 1. 2004 Adult cardiac surgery in New York State 2000-2002. Available at: https://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2000-2002_adult_cardiac_surgery.pdf. Accessed September 29, 2017. 2. 2017 Adult cardiac surgery in New York State 2012-2014. Available at: https://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2012-2014_adult_cardiac_surgery.pdf. Accessed September 29, 2017.
      Table E3Major adjudication changes
      YearTotal isolated CABGCardiogenic shockEmergent or emergent-salvageMI <24 h
      Submitted% of CABGChangedSubmitted% of CABGChangedSubmitted% of CABGChanged
      20073396752.236 (48.0)1614.732 (19.9)1474.331 (21.1)
      20083336852.532 (37.6)1594.827 (17.0)1394.222 (15.8)
      20093284521.629 (55.8)1033.111 (10.7)962.98 (8.3)
      20103169361.121 (58.3)872.710 (11.5)862.77 (8.1)
      20112840391.415 (38.5)993.511 (11.1)873.112 (13.8)
      20122680291.17 (24.1)933.58 (8.6)
      20132941351.26 (17.1)933.214 (15.1)
      20143063250.89 (36.0)953.112 (12.6)
      Total isolated CABGPeripheral vascular diseaseIsolated CABG
      Submitted% of CABGChangedSubmitted as CABG + OtherChanged to isolated CABG
      2007339614993 (62.4)
      2008333613689 (65.4)
      2009328414388 (61.5)
      2010316917462 (35.6)
      2011284013344 (33.1)
      2012268041215.471 (17.2)7531 (41.3)
      2013294140913.936 (8.8)7039 (55.7)
      2014306343814.368 (15.5)9855 (56.1)
      Values are presented as n, or %, or n (%). MI, Myocardial infarction; CABG, coronary artery bypass grafting.
      Table E4Risk factor odds ratios and relative change in odds, Massachusetts (MA) relative to national
      Risk factorModel parameterOdds ratio (99% confidence interval)
      DiabetesYear1.0452 (1.044-1.0464)
      InsulinYear1.0577 (1.056-1.0595)
      Moderate/severe chronic lung diseaseYear1.0173 (1.0154-1.0193)
      Morbid obeseYear1.0244 (1.0228-1.026)
      Cardiogenic shockYear1.0012 (0.9971-1.0053)
      Peripheral vascular diseaseYear0.9899 (0.9883-0.9914)
      Cerebrovascular diseaseYear1.0123 (1.0107-1.0139)
      ReoperationYear0.9459 (0.9425-0.9493)
      Emergent/salvageYear0.9993 (0.9967-1.0019)
      Ejection fraction <30%Year0.9982 (0.9958-1.0006)
      Ejection fraction 30%-39%Year0.9854 (0.9835-0.9873)
      Preoperative dialysisYear1.0681 (1.0641-1.0722)
      Age 65-74 yYear1.0142 (1.013-1.0154)
      Age ≥75 yYear0.9806 (0.9792-0.982)
      FemaleYear0.9884 (0.9871-0.9896)
      DiabetesMA1.1163 (1.0653-1.1696)
      InsulinMA1.041 (1.0023-1.0813)
      Moderate/severe chronic lung diseaseMA0.6302 (0.5719-0.6944)
      Morbid obeseMA0.8608 (0.8294-0.8934)
      Cardiogenic shockMA0.6637 (0.5322-0.8277)
      Peripheral vascular diseaseMA1.2153 (1.1438-1.2913)
      Cerebrovascular diseaseMA1.0159 (0.9791-1.0541)
      ReoperationMA0.5062 (0.4586-0.5588)
      Emergent/SalvageMA0.5665 (0.5237-0.6129)
      Ejection fraction <30%MA1.2662 (1.1598-1.3823)
      Ejection fraction 30%-39%MA1.1241 (1.045-1.2093)
      Preoperative dialysisMA0.8188 (0.7435-0.9018)
      Age 65-74 yMA1.0062 (0.9788-1.0343)
      Age ≥75 yMA1.2786 (1.2409-1.3175)
      FemaleMA0.8768 (0.8316-0.9244)
      DiabetesMA × Year0.9789 (0.9716-0.9864)
      InsulinMA × Year1 (1-1)
      Moderate/severe chronic lung diseaseMA × Year0.9708 (0.9553-0.9866)
      Morbid obeseMA × Year1 (1-1)
      Cardiogenic shockMA × Year0.9053 (0.8671-0.9451)
      Peripheral vascular diseaseMA × Year0.9704 (0.9605-0.9805)
      Cerebrovascular diseaseMA × Year1 (1-1)
      ReoperationMA × Year1 (1-1)
      Emergent/salvageMA × Year1 (1-1)
      Ejection fraction <30%MA × Year0.9648 (0.9504-0.9795)
      Ejection fraction 30%-39%MA × Year0.9817 (0.9695-0.994)
      Preoperative dialysisMA × Year1 (1-1)
      Age 65-74 yMA × Year1 (1-1)
      Age ≥75 yMA × Year1 (1-1)
      FemaleMA × Year0.9913 (0.9826-1)
      MA, Massachusetts.

      References

        • Shahian D.M.
        • Edwards F.H.
        • Jacobs J.P.
        • Prager R.L.
        • Normand S.L.
        • Shewan C.M.
        • et al.
        Public reporting of cardiac surgery performance: part 1-history, rationale, consequences.
        Ann Thorac Surg. 2011; 92: S2-S11
        • Shahian D.M.
        • Edwards F.H.
        • Jacobs J.P.
        • Prager R.L.
        • Normand S.L.
        • Shewan C.M.
        • et al.
        Public reporting of cardiac surgery performance: part 2-implementation.
        Ann Thorac Surg. 2011; 92: S12-S23
        • Shahian D.M.
        • Normand S.L.
        • Friedberg M.W.
        • Hutter M.M.
        • Pronovost P.J.
        Rating the raters: the inconsistent quality of health care performance measurement.
        Ann Surg. 2016; 264: 36-38
        • Shahian D.M.
        • Jacobs J.P.
        • Badhwar V.
        • D'Agostino R.S.
        • Bavaria J.E.
        • Prager R.L.
        Risk aversion and public reporting. part 1: observations from cardiac surgery and interventional cardiology.
        Ann Thorac Surg. 2017; 104: 2093-2101
        • Shahian D.M.
        • Jacobs J.P.
        • Badhwar V.
        • D'Agostino R.S.
        • Bavaria J.E.
        • Prager R.L.
        Risk aversion and public reporting. Part 2: mitigation strategies.
        Ann Thorac Surg. 2017; 104: 2102-2110
        • Hannan E.L.
        • Cozzens K.
        • King III, S.B.
        • Walford G.
        • Shah N.R.
        The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes.
        J Am Coll Cardiol. 2012; 59: 2309-2316
        • Schneider E.C.
        • Epstein A.M.
        Use of public performance reports: a survey of patients undergoing cardiac surgery.
        JAMA. 1998; 279: 1638-1642
        • Schneider E.C.
        • Epstein A.M.
        Influence of cardiac-surgery performance reports on referral practices and access to care—a survey of cardiovascular specialists.
        N Engl J Med. 1996; 335: 251-256
        • Harlan B.J.
        Statewide reporting of coronary artery surgery results: a view from California.
        J Thorac Cardiovasc Surg. 2001; 121: 409-417
        • Shahian D.M.
        • Grover F.L.
        • Prager R.L.
        • Edwards F.H.
        • Filardo G.
        • O'Brien S.M.
        • et al.
        The Society of Thoracic Surgeons voluntary public reporting initiative: the first 4 years.
        Ann Surg. 2015; 262: 526-535
        • Shahian D.M.
        • Jacobs J.P.
        • Edwards F.H.
        • Brennan J.M.
        • Dokholyan R.S.
        • Prager R.L.
        • et al.
        The Society of Thoracic Surgeons National database.
        Heart. 2013; 99: 1494-1501
        • Shahian D.M.
        • Jacobs J.P.
        Health services information: lessons learned from the Society of Thoracic Surgeons National database.
        in: Sobolev B. Levy A. Goring S. Data and measures in health services research. Springer US, Boston2016: 1-24
        • Massachusett Department of Health
        Cardiac services and treatments.
        (Available at:)
        • Green J.
        • Wintfeld N.
        Report cards on cardiac surgeons. Assessing New York State's approach.
        N Engl J Med. 1995; 332: 1229-1232
        • Chassin M.R.
        • Hannan E.L.
        • DeBuono B.A.
        Benefits and hazards of reporting medical outcomes publicly.
        N Engl J Med. 1996; 334: 394-398
        • The Society of Thoracic Surgeons
        Homepage.
        (Available at:)
        www.sts.org
        Date accessed: January 8, 2019
      1. The STS National database.
        (Available at:)
        • Massachusetts Data Analysis Center
        Adult coronary artery bypass graft surgery FY14 annual report.
        (Available at:)
        • Barringhaus K.G.
        • Zelevinsky K.
        • Lovett A.
        • Normand S.L.
        • Ho K.K.
        Impact of independent data adjudication on hospital-specific estimates of risk-adjusted mortality following percutaneous coronary interventions in massachusetts.
        Circ Cardiovasc Qual Outcomes. 2011; 4: 92-98
        • Shahian D.M.
        • O'Brien S.M.
        • Filardo G.
        • Ferraris V.A.
        • Haan C.K.
        • Rich J.B.
        • et al.
        The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery.
        Ann Thorac Surg. 2009; 88: S2-S22
        • O'Brien S.M.
        • Shahian D.M.
        • Filardo G.
        • Ferraris V.A.
        • Haan C.K.
        • Rich J.B.
        • et al.
        The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2--isolated valve surgery.
        Ann Thorac Surg. 2009; 88: S23-S42
        • Shahian D.M.
        • O'Brien S.M.
        • Filardo G.
        • Ferraris V.A.
        • Haan C.K.
        • Rich J.B.
        • et al.
        The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3—valve plus coronary artery bypass grafting surgery.
        Ann Thorac Surg. 2009; 88: S43-S62
        • Shahian D.M.
        • Jacobs J.P.
        • Badhwar V.
        • Kurlansky P.A.
        • Furnary A.P.
        • Cleveland Jr., J.C.
        • et al.
        The Society of Thoracic Surgeons 2018 adult cardiac surgery risk models: part 1-background, design considerations, and model development.
        Ann Thorac Surg. 2018; 105: 1411-1418
        • O'Brien S.M.
        • Feng L.
        • He X.
        • Xian Y.
        • Jacobs J.P.
        • Badhwar V.
        • et al.
        The Society of Thoracic Surgeons 2018 adult cardiac surgery risk models: part 2-statistical methods and results.
        Ann Thorac Surg. 2018; 105: 1419-1428
        • O'Brien S.M.
        • Shahian D.M.
        • DeLong E.R.
        • Normand S.L.
        • Edwards F.H.
        • Ferraris V.A.
        • et al.
        Quality measurement in adult cardiac surgery: part 2—statistical considerations in composite measure scoring and provider rating.
        Ann Thorac Surg. 2007; 83: S13-S26
        • Ho V.
        • Ku-Goto M.H.
        • Jollis J.G.
        Certificate of Need (CON) for cardiac care: controversy over the contributions of CON.
        Health Serv Res. 2009; 44: 483-500
        • DiSesa V.J.
        • O'Brien S.M.
        • Welke K.F.
        • Beland S.M.
        • Haan C.K.
        • Vaughan-Sarrazin M.S.
        • et al.
        Contemporary impact of state certificate-of-need regulations for cardiac surgery: an analysis using the Society of Thoracic Surgeons' National Cardiac Surgery Database.
        Circulation. 2006; 114: 2122-2129
        • Birkmeyer J.D.
        • Stukel T.A.
        • Siewers A.E.
        • Goodney P.P.
        • Wennberg D.E.
        • Lucas F.L.
        Surgeon volume and operative mortality in the United States.
        N Engl J Med. 2003; 349: 2117-2127
        • Birkmeyer J.D.
        • Siewers A.E.
        • Finlayson E.V.A.
        • Stukel T.A.
        • Lucas F.L.
        • Batista I.
        • et al.
        Hospital volume and surgical mortality in the United States.
        N Engl J Med. 2002; 346: 1128-1137
        • Shahian D.M.
        • Normand S.L.
        The volume-outcome relationship: from Luft to Leapfrog.
        Ann Thorac Surg. 2003; 75: 1048-1058
        • Burke L.G.
        • Frakt A.B.
        • Khullar D.
        • Orav E.J.
        • Jha A.K.
        Association between teaching status and mortality in US Hospitals.
        JAMA. 2017; 317: 2105-2113
        • Shahian D.M.
        • Liu X.
        • Meyer G.S.
        • Torchiana D.F.
        • Normand S.L.
        Hospital teaching intensity and mortality for acute myocardial infarction, heart failure, and pneumonia.
        Med Care. 2014; 52: 38-46
        • Waldo S.W.
        • McCabe J.M.
        • O'Brien C.
        • Kennedy K.F.
        • Joynt K.E.
        • Yeh R.W.
        Association between public reporting of outcomes with procedural management and mortality for patients with acute myocardial infarction.
        J Am Coll Cardiol. 2015; 65: 1119-1126
        • McCabe J.M.
        • Waldo S.W.
        • Kennedy K.F.
        • Yeh R.W.
        Treatment and outcomes of acute myocardial infarction complicated by shock after public reporting policy changes in New York.
        JAMA Cardiol. 2016; 1: 648-654
        • Joynt K.E.
        • Blumenthal D.M.
        • Orav E.J.
        • Resnic F.S.
        • Jha A.K.
        Association of public reporting for percutaneous coronary intervention with utilization and outcomes among Medicare beneficiaries with acute myocardial infarction.
        JAMA. 2012; 308: 1460-1468
        • McCabe J.M.
        • Joynt K.E.
        • Welt F.G.
        • Resnic F.S.
        Impact of public reporting and outlier status identification on percutaneous coronary intervention case selection in Massachusetts.
        JACC Cardiovasc Interv. 2013; 6: 625-630
        • Chassin M.R.
        Achieving and sustaining improved quality: lessons from New York State and cardiac surgery.
        Health Aff (Millwood). 2002; 21: 40-51
        • Jha A.K.
        • Epstein A.M.
        The predictive accuracy of the New York State coronary artery bypass surgery report-card system.
        Health Aff (Millwood). 2006; 25: 844-855
        • Mukamel D.B.
        • Weimer D.L.
        • Mushlin A.I.
        Interpreting market share changes as evidence for effectiveness of quality report cards.
        Med Care. 2007; 45: 1227-1232
        • Mukamel D.B.
        • Mushlin A.I.
        The impact of quality report cards on choice of physicians, hospitals, and HMOs: a midcourse evaluation.
        Jt Comm J Qual Improv. 2001; 27: 20-27
        • Mukamel D.B.
        • Mushlin A.I.
        Quality of care information makes a difference: an analysis of market share and price changes after publication of the New York State Cardiac Surgery Mortality Reports.
        Med Care. 1998; 36: 945-954

      Linked Article

      • Commentary: Massachusetts exceptionalism
        The Journal of Thoracic and Cardiovascular SurgeryVol. 158Issue 1
        • Preview
          Mandatory public reporting has been generally unpopular since it began in New York in 1991, and it remains controversial today. Opponents emphasize a variety of purported evils (gaming, shaming, risk-aversion, meaningless distinctions, etc). Shahian and colleagues1 show that outcomes for isolated coronary artery bypass grafting procedures in Massachusetts improved between 2003 and 2014 and were significantly better than the outcomes reflected in the Society of Thoracic Surgeons database over the same period.
        • Full-Text
        • PDF
        Open Archive