If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Beginning in 2002, all 14 Massachusetts nonfederal cardiac surgery programs submitted Society of Thoracic Surgeons (STS) National Database data to the Massachusetts Data Analysis Center for mandatory state-based analysis and reporting, and to STS for nationally benchmarked analyses. We sought to determine whether longitudinal prevalences and trends in risk factors and observed and expected mortality differed between Massachusetts and the nation.
Methods
We analyzed 2003 to 2014 expected (STS predicted risk of operative [in-hospital + 30-day] mortality), observed, and risk-standardized isolated coronary artery bypass graft mortality using Massachusetts STS data (N = 39,400 cases) and national STS data (N = 1,815,234 cases). Analyses included percentage shares of total Massachusetts coronary artery bypass graft volume and expected mortality rates of 2 hospitals before and after outlier designation.
Results
Massachusetts patients had significantly higher odds of diabetes, peripheral vascular disease, low ejection fraction, and age ≥75 years relative to national data and lower odds of shock (odds ratio, 0.66; 99% confidence interval, 0.53-0.83), emergency (odds ratio, 0.57, 99% confidence interval, 0.52-0.61), reoperation, chronic lung disease, dialysis, obesity, and female sex. STS predicted risk of operative [in-hospital + 30-day] mortality for Massachusetts patients was higher than national rates during 2003 to 2007 (P < .001) and no different during 2008 to 2014 (P = .135). Adjusting for STS predicted risk of operative [in-hospital + 30-day] mortality, Massachusetts patients had significantly lower odds (odds ratio, 0.79; 99% confidence interval, 0.66-0.96) of 30-day mortality relative to national data. Outlier programs experienced inconsistent, transient influences on expected mortality and their percentage shares of Massachusetts coronary artery bypass graft cases.
Conclusions
During 12 years of mandatory public reporting, Massachusetts risk-standardized coronary artery bypass graft mortality was consistently and significantly lower than national rates, expected rates were comparable or higher, and evidence for risk aversion was conflicting and inconclusive.
Operative mortality, Society of Thoracic Surgeons (STS) national data (excluding Massachusetts) versus STS data exclusively from Massachusetts and Massachusetts Data Analysis Center (Mass-DAC).
From 2003-2014, risk-adjusted CABG outcomes in Massachusetts, a public reporting state, were consistently superior to STS national results; evidence for risk aversion was conflicting and inconclusive.
During 12 years of mandatory public reporting (2003-2014), Massachusetts risk-adjusted CABG mortality was consistently lower than national rates, expected mortality was comparable or higher, and evidence for risk aversion was inconclusive. These nationally benchmarked results support continued study of public reporting as a mechanism to educate patients and potentially to improve health care outcomes.
Public reporting of hospital-specific outcomes by government and commercial organizations remains highly controversial. Proponents cite the need for transparency and accountability, the ethical obligation to facilitate informed decision making by consumers and referring physicians, and the ability of report cards to stimulate performance improvement. Detractors are concerned about the accuracy and effectiveness of report cards and their unintended consequences such as gaming, risk aversion, and reduced access to care.
The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes.
The Massachusetts cardiac surgery public reporting program provides a unique natural laboratory in which to study these issues. State legislation mandates the use of the Society of Thoracic Surgeons (STS) Adult Cardiac Surgery Database (ACSD) by all cardiac surgery programs, which ensures a highly representative, audited, and risk-adjusted national comparator for Massachusetts, using identical data specifications.
Beginning January 2002, Massachusetts Department of Public Health (DPH) required collection of clinical and vital status data from all 14 nonfederal Massachusetts cardiac surgery programs, stipulating participation in the STS ACSD
and parallel submission of fully identified data to the Massachusetts Data Analysis Center (Mass-DAC). Reports of hospital outcomes for isolated coronary artery bypass grafting (CABG) were published for 2002 to 2014 procedures, and surgeon outcomes for 2002 to 2010 procedures.
Because of common STS ACSD data platforms, cross-sectional and longitudinal analyses of cardiac surgical practice in Massachusetts versus STS national data are facilitated. We leverage this to compare Massachusetts and STS isolated CABG volumes; expected, observed, and risk-standardized outcomes; and risk factor prevalences. The value of peer coding adjudication in Massachusetts and the consequences of outlier designation were also studied.
Methods
Study Populations
We excluded 2002 data due to inaugural year coding issues, similar to early New York reports.
We analyzed 3 populations of adults aged 18 years or older undergoing isolated CABG between January 1, 2003, and December 31, 2014: isolated CABG cases submitted to STS from Massachusetts hospitals (MA [STS] cohort, n = 39,400), isolated CABG procedures submitted to STS from US states, excluding Massachusetts (national [STS] cohort, n = 1,815,234), and isolated CABG cases in all nonfederal Massachusetts hospitals, submitted to Mass-DAC (n = 40,655) for the Massachusetts public reporting program. The additional Mass-DAC cases compared with cases submitted to STS reflects slight differences in isolated CABG case inclusion criteria between the 2 sources.
Data Sources
We used information publicly reported on the Massachusetts DPH
previously published Mass-DAC reports, and additional data provided by these organizations.
Patient and Hospital Outcomes
Mass-DAC reports observed, expected, and risk-standardized 30-day all-cause mortality. STS reports observed and expected operative mortality, which includes all deaths during the index hospitalization, regardless of timing, and all deaths within 30 days regardless of location. In addition to these standard reports, we extracted STS 30-day observed mortality and Mass-DAC operative mortality from original data sources.
Data Collection and Quality Assurance
STS cohort
STS ACSD data specifications, cleaning, and quality assurance processes are published elsewhere.
Annual external audits of 10% of participating programs demonstrate 95% to 97% overall agreement rates. During the study period, multiple STS ACSD versions (2.35-2.81) were used depending on year.
Data submitted to STS were missing <1% for most variables, presumably because Massachusetts data managers were required to have complete data for their contemporaneous Mass-DAC submissions. Patients with missing or unknown operative mortality status were assumed alive, consistent with STS coding practice during the study period.
Mass-DAC cohort
Mass-DAC data collection, cleaning, and quality assurance processes are published elsewhere.
Mass-DAC reports information assembled from several linked data sources. Patient-specific risk factor and outcome data are collected by trained hospital personnel using standard STS ACSD data submission forms.
External administrative data resources and medical record review are used to validate submitted data and to ascertain or verify 30-day vital status (Table E1). Missing data were extremely rare.
Before analyses, Mass-DAC charts coded for selected risk factors (routinely: shock, emergent or emergent/salvage status, myocardial infarction <24 hours, and, beginning 2012, peripheral vascular disease [PVD]; periodically, low ejection fraction [EF], moderate/severe chronic lung disease, or dialysis) were adjudicated by surgeons and data managers from all 14 programs, as previously described for Mass-DAC percutaneous coronary intervention (PCI) public reporting.
Impact of independent data adjudication on hospital-specific estimates of risk-adjusted mortality following percutaneous coronary interventions in massachusetts.
and clinical review of medical records and operative notes, all procedures coded “CABG + other” were also adjudicated, based on the magnitude and incremental risk of the “other” procedure, to confirm they were not isolated CABG cases that were intentionally or inadvertently coded as combined procedures (that are not publicly reported).
Expected Mortality and Outlier Determination
STS cohort
For operational feedback reports, STS recalibrates its risk models to achieve a ratio of observed to expected mortality (O/E) of 1 and periodically estimates completely new models.
Because of frequent re-estimation and only 14 programs, annual Massachusetts O/E ratios are typically close to unity.
Fully Bayesian hierarchical logistic (surgery years 2002-2009) and Poisson (surgical procedures beginning in 2010) regression models are estimated that include hospital-specific random intercepts.
Hospitals were categorized as outliers if their standardized mortality incidence rate 95% credible interval excluded the statewide average, or if their observed mortality rate was statistically different from that predicted by all other hospitals (ie, with the problematic hospital's data excluded from risk model development).
The Massachusetts Cardiac Care Outlier Committee (composed of surgeons from the Massachusetts STS) reviewed Mass-DAC annual results before final release, including clinical information from statistical outliers, to determine whether there were systemic versus unique case mix issues to explain low performance.
Statistical Analyses
We compared longitudinal trends in Massachusetts isolated CABG volume with trends from STS (national) and from New York, another public reporting state.
Expected and adjusted mortality rate analyses comparing Massachusetts and national [STS] cohorts used MA [STS] data (similar to Mass-DAC data) and STS predicted risk of operative mortality (PROM). Risk factor prevalence comparisons primarily used peer-adjudicated Mass-DAC data, supplemented by MA [STS] data.
We analyzed annual observed operative and 30-day mortality rates for all 3 study cohorts; expected operative mortality rates and O/E ratios for national [STS] and MA [STS] cohorts, using STS risk models; and expected 30-day mortality for the Mass-DAC cohort.
To statistically compare national [STS] and Massachusetts risk factor prevalences, surgical outcomes, and their corresponding longitudinal changes, we estimated separate logistic regression models for 15 selected risk factors and for observed operative and 30-day mortality. We linked the log-odds of each end point (ie, risk factor or mortality) to categorical variables indicating Massachusetts (Mass-DAC data source for risk factors, MA [STS] data source for mortality) versus national [STS] data, year of surgery (year 0 [2003] to year 11 [2014]), and an interaction of Massachusetts indicator and year; for mortality, these regressions were also estimated with adjustment for STS PROM, a summary risk score based on patient factors. Significant odds ratios (ORs) for interaction terms imply that annual rates of change in mortality or risk factor odds differed between Massachusetts and national [STS] data; when interaction terms are nonsignificant, the MA indicator represents the overall OR between Massachusetts and national [STS], and the year indicator is the overall rate of change. We noted if annual trends were increasing or decreasing in both cohorts, or directionally opposite. To minimize family-wise error rate, we used conservative 99% confidence intervals.
We explored hospital-specific percentage shares of Massachusetts CABG volumes between 2003 and 2014, changes in the share of CABG cases at 2 outlier hospitals, and expected mortality rate trends of outlier hospitals compared with the volume weighted average expected mortality (Mass-DAC models) for all Massachusetts hospitals.
Mass-DAC used hospital-aggregated, de-identified data. Raw data and procedures associated with assessing hospital quality were reviewed by the Harvard Medical School Institutional Review Board. STS data were likewise de-identified, and STS data collection and analysis procedures were reviewed by the Duke Institutional Review Board.
Results
CABG Volumes
Massachusetts isolated CABG admissions declined 33.5% from 2002 (n = 4603) to 2014 (n = 3063), less than the 56.1% decrease in comparable STS procedures/program nationally, the 52.7% decrease among 388 sites participating continuously in the STS ACSD from 2002 to 2014, and the 50.7% decrease in New York (Table E2).
Mass-DAC Verification of Case Completeness and Deaths
Between 2004 and 2014, Mass-DAC verification of case completeness and vital status resulted in 0 to 2 (mean, 0.9) additional isolated CABG procedures (0.03% of all cases) and 0 to 5 (mean, 2) net 30-day mortalities (0.06% of all cases) added annually to data submitted by hospitals.
Mass-DAC Adjudication
From 2002 to 2014, 541 to 781 total CABG cases (mean, 636.8) were reviewed annually for various indications, including 273 to 724 risk variable (mean, 445.8) audits. Excluding 2002, 38-138 (mean, 82.7) risk variables annually had coding changes.
Detailed annual adjudication data are available from 2007 to 2014 (Table E3) for high-risk variables and to substantiate CABG + other procedure designation. Compared with the 9.8% decline in isolated CABG volume during this period, cases submitted with shock or emergent/emergent-salvage status declined substantially more (eg, shock, 67% decrease; emergent/emergent-salvage, 41% decrease), as did their submission rates as percentages of isolated CABG cases and their rates of coding changes.
Between 2007 and 2014, 70 to 174 (mean, 122) procedures annually were coded by hospitals as CABG + other. Both the number (149 to 98) of submitted CABG + other cases and the number changed to isolated CABG (93 to 55) decreased between 2007-2014; overall, between 33.1% and 65.4% CABG + other cases were reclassified isolated CABG.
Mass-DAC Versus STS Risk Factor Prevalence
Massachusetts risk factor prevalences (ie, Mass-DAC and MA [STS]) are compared with national [STS] prevalences between 2003 and 2014 in Table 1 and Figure E1, Figure E2, Figure E3, Figure E4, Figure E5. Mass-DAC and MA [STS] rates closely agreed for most, but not all (eg, several peer-adjudicated variables) risk factors.
Table 1Risk factor prevalences, Massachusetts versus national [STS]
Year
Mean age (y)
Age 65- 74 y (%)
Age ≥75 y (%)
Female (%)
EF 0.30- 0.39 (%)
EF < 0.30 (%)
CVD (%)
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
2003
65.0
66.7
31.6
32.1
21.8
26.8
27.9
26.5
10.7
12.4
6.4
7.7
13.5
14.0
2004
65.1
66.9
31.5
32.2
22.2
27.8
27.9
25.5
10.4
11.0
6.3
8.4
14.0
14.8
2005
65.1
66.4
31.3
31.2
22.0
26.0
27.2
23.6
10.5
10.9
6.4
8.1
13.7
14.1
2006
65.0
66.3
31.5
31.3
21.5
25.6
27.3
24.4
10.1
9.9
6.3
6.3
13.7
13.7
2007
64.9
66.0
31.3
30.7
21.0
25.1
27.1
24.3
10.1
10.9
6.3
6.8
13.7
14.0
2008
64.8
66.0
31.8
31.8
20.4
24.5
27.3
22.4
9.6
10.1
6.2
5.5
14.2
12.9
2009
64.8
66.1
32.4
33.0
20.0
23.8
26.9
22.5
9.4
9.1
6.2
6.4
14.3
14.8
2010
64.8
65.8
32.7
32.1
19.9
23.0
26.3
22.9
9.3
9.4
6.0
6.2
14.5
13.3
2011
64.9
66.2
32.9
32.4
19.7
23.9
26.4
22.4
9.3
9.9
6.1
5.8
14.1
15.4
2012
64.8
66.1
33.8
35.1
19.2
22.1
26.1
22.2
9.5
9.1
6.2
5.5
14.1
14.2
2013
65.0
66.3
34.6
32.7
19.4
24.5
25.7
22.4
9.4
8.8
6.3
5.8
14.2
13.9
2014
64.9
66.5
34.6
36.3
18.9
23.5
25.1
22.0
9.4
8.3
6.4
5.9
16.5
17.2
Dialysis (%)
Morbid obesity (%)
Reoperation (%)
Moderate/severe CLD (%)
Diabetes (%)
Insulin dependent (%)
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
STS
Mass-DAC
2003
1.5
1.3
14.3
12.0
9.0
6.0
35.8
38.1
10.2
12.2
2004
1.6
1.4
14.3
12.0
5.3
2.5
9.2
5.6
36.2
37.0
10.5
10.6
2005
1.7
1.2
14.9
12.8
4.8
2.6
9.5
5.5
36.8
39.3
11.2
11.7
2006
1.8
1.8
15.3
13.7
4.6
2.2
9.5
5.9
37.6
40.0
11.4
12.5
2007
1.9
1.9
15.8
14.3
4.4
2.6
9.6
5.8
39.1
41.8
12.1
14.4
2008
2.4
2.1
16.3
13.4
4.1
2.7
9.9
6.3
39.7
37.3
12.9
11.8
2009
2.5
2.4
16.7
15.0
3.9
1.7
10.1
5.1
40.5
39.3
13.6
12.9
2010
2.6
2.3
16.8
15.2
3.8
1.8
10.4
5.8
41.4
39.7
14.1
14.6
2011
2.8
2.0
17.1
16.1
3.6
2.1
10.4
5.8
43.1
41.5
15.0
15.1
2012
2.8
1.6
17.4
15.8
3.4
1.3
10.7
4.8
44.9
43.1
15.9
15.9
2013
2.8
2.2
17.3
16.2
3.1
1.4
10.7
4.7
45.7
44.5
16.9
17.4
2014
3.0
2.3
17.7
15.6
2.9
1.4
10.1
5.6
47.3
43.8
17.5
17.6
Cardiogenic shock (%)
Emergent ± salvage (%)
MI <24 h (%)
PVD (%)
STS
Mass-DAC
MA [STS]
STS
Mass-DAC
MA [STS]
STS
Mass-DAC
MA [STS]
STS
Mass-DAC
MA [STS]
2003
1.9
1.6
2.1
4.6
3.0
3.8
3.9
3.1
3.1
15.7
17.4
17.4
2004
1.9
1.1
2.2
4.8
3.1
4.5
4.2
3.8
4.2
15.4
17.7
17.8
2005
2.0
1.0
1.9
5.0
2.5
3.4
4.5
2.9
3.6
15.5
17.5
17.3
2006
2.0
0.8
1.7
5.0
2.7
3.5
4.6
3.7
4.2
15.3
17.7
17.7
2007
2.0
0.9
1.6
4.9
3.4
4.2
4.7
3.1
4.1
15.1
17.1
17.1
2008
2.0
0.7
1.4
4.7
3.0
3.3
4.4
3.3
3.7
14.8
14.1
14.0
2009
1.9
0.6
1.1
4.6
2.4
2.6
4.3
2.4
2.7
15.0
15.3
15.2
2010
2.0
0.5
1.0
4.5
2.5
2.8
4.3
2.6
2.8
15.1
15.5
15.5
2011
2.0
0.4
0.6
4.7
2.6
2.9
4.2
2.7
2.8
14.6
15.5
15.8
2012
2.0
0.7
0.6
5.0
2.5
2.6
4.0
2.4
2.3
14.2
11.8
14.1
2013
1.8
0.8
0.9
4.8
2.6
3.2
3.9
2.8
2.4
14.1
12.6
14.1
2014
2.0
0.6
0.8
4.8
2.5
2.6
3.6
2.5
2.2
14.4
12.5
14.5
For 4 risk factors that underwent peer-adjudication in Massachusetts, Table 1 includes prevalences of those factors in data submitted directly from Massachusetts programs to the Society of Thoracic Surgeons (STS) national database (MA [STS]), as well as the comparable prevalences in Massachusetts Data Analysis Center (Mass-DAC) data following peer-adjudication. EF, Ejection fraction; CVD, cerebrovascular disease; STS, Society of Thoracic Surgeons; DAC, Data Analysis Center; CLD, chronic lung disease; MI, myocardial infarction; PVD, peripheral vascular disease.
Regression analysis results of selected risk factors are summarized in Figure 1 and Table E4. Relative to national [STS] patients, adjudicated Mass-DAC data showed significantly lower odds of cardiogenic shock, reoperation, emergency/emergency-salvage, chronic lung disease, dialysis, morbid obesity, and female sex; and significantly greater odds of diabetes, PVD, low EF, age ≥75 years (Figure 1, B). Annual rates of change in risk factor odds in Massachusetts relative to national [STS] data were not statistically different for 8 of 15 risk factors studied. Odds of cardiogenic shock decreased significantly in Massachusetts relative to national [STS] rates, which were slightly increasing during the study period [Figure 1, C]). Odds of diabetes increased at a slightly slower rate in Massachusetts than nationally, whereas odds of PVD, EF 0.30 to 0.39 and EF <0.30, and female sex fell faster in Massachusetts.
Figure 1Multivariable logistic regression analysis of selected risk factor odds, 2003 to 2014. To statistically compare national [STS] and Massachusetts risk factor prevalences and their longitudinal changes, we estimated separate logistic regression models for 15 selected risk factors. We linked the log-odds of each risk factor to categorical variables indicating Massachusetts Data Analysis Center (Mass-DAC) versus national STS data (excluding MA patients), year of surgery (year 0 [2003] to year 11 [2014]), and an interaction of Mass-DAC indicator and year. Significant odds ratios for interaction terms imply that annual rates of change in risk factor odds differed between Massachusetts and national [STS] data; when interaction terms are non-significant, the MA indicator represents the overall odds ratio between Massachusetts and national [STS], and the year indicator is the overall rate of change. We noted if annual trends were increasing or decreasing in both cohorts, or directionally opposite. Odds ratios with 99% confidence intervals (CI) are summarized for the 3 model co-variables: A, Massachusetts (MA) (Yes/No) = MA relative to national STS. B, Year = Annual change (2003 = 0 increasing to 2014 = 11). C, MA × Year (interaction term) = Annual change in MA relative to annual change in national STS. STS, Society of Thoracic Surgeons; Mod/Sev CLD, Moderate or severe chronic lung disease; PVD, peripheral vascular disease; CVD, cerebrovascular disease; EF, ejection fraction; Pre-op, preoperative.
Massachusetts observed operative (Figure 2, A) and 30-day (Figure E6) mortality rates were no different than national [STS] rates in 2003, then became significantly and consistently lower than STS national rates, although they increased after 2011. For several annual reports of operative and 30-day mortality, Mass-DAC observed mortality rates were slightly higher than MA [STS] rates, probably reflecting enhanced vital status verification.
Figure 2Expected and observed mortality rates. A, Observed operative mortality for the Society of Thoracic Surgeons (STS) national cohort (excluding Massachusetts (MA) patients), MA patient data submitted by hospitals directly to STS (MA [STS]), and Massachusetts data adjudicated and verified by the Massachusetts Data Analysis Center (Mass-DAC), which captured a few more deaths in certain years. B, Expected operative mortality rates (in-hospital + 30-day) for the national [STS] cohort and MA [STS] cohorts, based on STS data and STS 2008 risk models.
Using the STS isolated CABG risk model, MA [STS] PROM rates were higher than national [STS] rates from 2003 to 2007 (P < .001) but declining, compared with relatively flat national [STS] expected rates; from 2008-2014, MA [STS] and national [STS] PROM rates were similar (P = .135) and stable (Figure 2, B). Using STS data and risk models, MA [STS] O/E for operative mortality generally ranged from 0.6 to 0.8 (Figure E7).
Because Mass-DAC risk models for Massachusetts public reporting were derived only from Massachusetts programs and re-estimated annually, observed and expected 30-day mortality rates tracked very closely (Figure E8).
In regression analyses incorporating STS PROM, the MA [STS] CABG cohort had significantly lower adjusted odds of operative (OR, 0.82; 99% confidence interval, 0.69-0.98) and 30-day (OR, 0.79; 99% confidence interval, 0.66-0.96) mortality relative to the national [STS] cohort (Figure 3), with no significant difference in the annual change in MA [STS] mortality odds relative to the annual change in national [STS] cohort odds (MA × Year interaction term).
Figure 3Operative and 30-day mortality odds and relative change in odds, Massachusetts STS (MA [STS]) data versus national STS data (excluding MA patients). To statistically compare national and Massachusetts surgical outcomes and their longitudinal changes, we estimated logistic regression models for observed operative and 30-day mortality. We linked the log-odds of each end point to categorical variables indicating MA [STS] versus national [STS] data, year of surgery (Year: 0 [2003] to 11 [2014]), and an interaction of MA indicator and year. These regressions were also estimated with adjustment for STS predicted risk of mortality, a summary risk score based on patient factors. Significant odds ratios for interaction terms imply that annual rates of change in mortality differed between Massachusetts and national data; when interaction terms are nonsignificant, the MA indicator represents the overall odds ratio between Massachusetts and national data, and the year indicator is the overall rate of change. Multivariable logistic regression analysis results shown for operative (red) and 30-day (blue) mortality outcomes without (unadjusted, open symbols) and with (adjusted, closed symbols) risk-adjustment using 2008 STS predicted risk of operative mortality model. Covariates: MA (Yes/No) = MA relative to national STS; Year = annual change (2003 = 0 increasing to 2014 = 11); MA × Year (interaction term) = annual change in MA mortality odds relative to annual change in national STS mortality odds. OR, Odds ratio; 99% CI, 99% confidence interval.
Massachusetts provided surgeon-level reports for procedures performed between 2002 and 2010 (last report February 2012 based on fiscal year [FY] 2008-2010 data). Two high-mortality surgeon outliers were identified in calendar year (CY) 2002-2004 report (published 2006), 3 in the CY2003-CY2005 report (published in 2007), and 2 in the FY2003-FY2006 report (published in February 2008). Because of overlapping 3-year time windows, the same outliers were identified in several reports. There were only 3 unique outlier surgeons, all at outlier hospitals.
Consequences of Outlier Designation
Subsequent expected mortality rates
Two Massachusetts hospitals were transiently designated as 30-day high mortality outliers—Hospital A in 2005 and 2007 reports (2003 and 2005 procedures); and Hospital B in the 2006 report (2004 procedures). Before outlier designation, both hospitals had higher expected mortality rates than the volume weighted state average (Figure 4).
Figure 4Outlier designation and expected mortality rates. Before outlier designation, both hospitals had higher expected mortality rates than the volume weighted state average. As a result of its outlier designation, Hospital A temporarily suspended its cardiac program and made extensive, state-recommended changes. Although again an outlier 2 years later, their results were substantially improving. Hospital A's expected mortality decreased after its initial outlier designation; by 2010, its expected mortality rate had returned to the state average, which it tracked closely thereafter. Hospital B's expected mortality increased slightly following outlier designation, then converged towards and tracked the state mean from 2010-2011 onward.
Massachusetts DPH temporarily closed Hospital A's program after its first outlier designation, leading to extensive structural, personnel, and leadership changes. Although again an outlier 2 years later, their results were substantially improving. Hospital A's expected mortality decreased after its initial outlier designation, but by 2010 its expected mortality rate had returned to the state average, which it tracked closely thereafter (Figure 4).
Hospital B's expected mortality increased slightly following outlier designation, then converged toward and tracked the state mean from 2010 to 2011 onward.
Share of Massachusetts CABG Cases
Following initial outlier designation (CY2003 report, published in October 2005) and temporary closure, Hospital A's share of all Massachusetts CABG cases dropped from 6.5% (2005) to 4.9% (2006). With implementation of numerous changes recommended by a state-mandated panel of experts, outcomes improved and their percentage share of Massachusetts CABG cases rebounded to 7.1% in 2007, with steady increases thereafter despite their second identification as an outlier (October 2007, based on CY2005 outcomes) (Figure 5).
Figure 5Outlier designation and percentage share of total Massachusetts coronary artery bypass graft (CABG) cases. Following initial outlier designation and temporary closure, Hospital A's share of Massachusetts CABG cases decreased. After implementation of numerous changes, their outcomes improved and their percentage share of Massachusetts CABG cases rebounded in 2007, steadily increasing thereafter despite their second identification as an outlier. Hospital B experienced growth in its share of Massachusetts CABG volume the year following its outlier designation; this percentage then stabilized for 4 years and subsequently increased.
Hospital B experienced growth in its share of Massachusetts CABG volume from 4.4% to 4.7% the year following its outlier designation (CY2004 report, published October 2006); this percentage then stabilized for 4 years and subsequently increased.
Overall, both Hospitals A and B, despite their transient outlier status, experienced substantial increases in their percentage shares of Massachusetts CABG volume from 2003 to 2014 (18.7% and 37.1%, respectively) (Figure 6).
Figure 6Hospital-specific percentage share of total Massachusetts coronary artery bypass graft (CABG) cases, 2003 versus 2014. Between 2003 and 2014, 8 of 14 Massachusetts adult cardiac surgery programs experienced volume growth, including both programs that had been transient outliers. Relative volume shifts among hospitals were more related to initiation of 3 new cardiac surgery programs, which drew volume from existing programs, and with altered referral patterns at several hospitals due to leadership and staff changes.
Massachusetts observed mortality was significantly and consistently less than the STS PROM for its case-mix (O/E < 1), and adjusted mortality odds were about 20% lower than national STS results. These findings could be associated with public reporting, especially since Massachusetts and STS national rates were similar at the beginning of the public reporting experience (Figure 2, A). However, a direct causal association cannot be inferred from these data, and there are numerous other confounders. For example, Massachusetts also has a longstanding Determination of Need (referred to in other states as Certificate of Need) program, which some states use to restrict the number of hospitals offering certain high cost services
Contemporary impact of state certificate-of-need regulations for cardiac surgery: an analysis using the Society of Thoracic Surgeons' National Cardiac Surgery Database.
; in Massachusetts, this resulted in relatively fewer but higher volume cardiac surgery programs. Furthermore, most Massachusetts cardiac surgery units are based in major academic centers or their affiliates. Both higher volume and teaching intensity have been associated with improved health care outcomes.
and this is another potential explanation for lower observed mortality rates in Massachusetts. Risk aversion is typically described as the inappropriate denial of care to high-risk patients who may benefit most from interventions. Not fully trusting the adequacy of risk-adjustment, some providers may be concerned that the anticipated worse outcomes of very high-risk patients will be reflected in their report card scores, thereby harming their practices and reputations. Although the existence and extent of this phenomenon was vigorously debated in the early New York and Pennsylvania CABG public reporting experience, the overall evidence for risk aversion in cardiac surgery is inconclusive, although it is well documented in PCI for complicated acute myocardial infarction.
Association of public reporting for percutaneous coronary intervention with utilization and outcomes among Medicare beneficiaries with acute myocardial infarction.
In the current study, the lower odds of several important risk factors in Massachusetts (eg, shock, emergent/emergent-salvage, and reoperation) could reflect risk aversion as a potential explanation for the lower observed CABG mortality in Massachusetts compared with the rest of the United States. However, peer-adjudication of risk factors in Massachusetts likely contributed in part to these lower prevalences, both by educating surgeons and data managers regarding definitional nuances, and when necessary by downcoding of submitted data. Other more common risk factors (eg, advanced age, low EF, diabetes, and PVD) had higher or similar prevalences in Massachusetts; temporal trends for many risk factors were no different.
When compared with relatively stable national [STS] rates, decreasing Massachusetts expected mortality from 2003 to 2007 could also be interpreted as supporting risk aversion. However, using identical STS risk models, MA [STS] expected mortality rates were significantly higher than national [STS] rates for this period, although declining, and similar from 2008 onward, which argues against risk aversion. The timeline of Massachusetts public reporting provides useful context for these observations. Massachusetts reporting legislation was passed in 2000 and immediately energized the cardiac surgery community. Two years of anticipation preceded the first year (2002) in which data were actually collected and analyzed. In this context, the significantly higher expected mortality rates for Massachusetts (MA [STS]) compared with national [STS] during the early years of public reporting, when there was maximum hospital and surgeon anxiety) seems inconsistent with substantial risk averse behavior.
The increase in MA [STS] observed mortality after 2011 could simply represent regression to the mean after exceptionally low 2011 observed rates. However, there was also a temporal association with phasing out of surgeon-level reporting during the preceding years, which some would argue made surgeons less reluctant to accept very high-risk patients.
Overall, our empirical findings are conflicting and inconclusive regarding risk aversion, at least at the aggregate level. Available Massachusetts data provide no insights regarding individual patient-level decisions, although anecdotal evidence suggests this phenomenon does sometimes occur, even if not at a detectable frequency within the CABG population.
Surgeon and data manager involvement with and support of the Massachusetts public reporting program were critical to its success. For example, Massachusetts’ peer adjudication provided additional coding review beyond standard STS audit processes and could be a model for other states and regions. Over time, progressive decreases in cases submitted with selected high-risk preoperative variables, and fewer changes in these adjudicated variables and in cases coded CABG + other, suggest the educational value of this exercise. The greatest divergence between adjudicated Mass-DAC risk factor coding and data submitted directly to STS occurred in the early years of Massachusetts public reporting. This probably reflects the initial coding inexperience of Massachusetts data managers and surgeons (most of whom had not previously participated in the STS Database) as well as some less precise STS variable definitions in earlier years. The Massachusetts peer adjudication process educated all stakeholders regarding optimal coding practices and nuances. It also made them aware that their coding would be adjudicated by peers, making intentional upcoding less likely. Contemporaneously, national STS resources to assist data managers with coding issues were expanded, and STS external audit processes became more robust. In the 2016 external national audit report of the STS ACSD, auditors agreed with submitted coding of cardiogenic shock, myocardial infarction timing, and status (ie, elective, urgent, emergent, or emergent/salvage) in 99.68%, 97.59%, and 97.63% of charts audited, respectively. Thus, because of both Massachusetts and national initiatives, coding for high-risk variables in these 2 data sources progressively converged and became more accurate, although there is a persistently lower prevalence of several such variables in Massachusetts.
Mass-DAC case completeness and mortality verification confirm that hospitals were not hiding cases or deaths; they demonstrate the utility of linking clinical registry data to state and national administrative sources for external validation.
The tendency for outliers to become risk averse, previously documented for PCI,
was transient and inconsistent in our study. Within a few years after outlier designation, expected mortality rates at both Massachusetts outlier programs had stabilized around the state average.
Quality of care information makes a difference: an analysis of market share and price changes after publication of the New York State Cardiac Surgery Mortality Reports.
only 1 of the 2 outlier hospitals experienced a negative influence on its share of Massachusetts CABG cases and that effect was transient. Many factors besides objective outcomes influence patient selection of a provider, including the recommendations of the referring physicians, family, and friends; geographic preferences; differences in copays; and limited choices available in their insurance plan. Relative volume shifts among hospitals were more related to initiation of 3 new cardiac surgery programs, which drew volume from existing programs, and with altered referral patterns at several hospitals due to leadership and staff changes. If underlying causes of outlier designation are addressed quickly and effectively, referral impact should be limited and transient.
Limitations
One major motivation for the Commonwealth of Massachusetts to mandate participation in the STS National Database was the prior lack of credible data regarding cardiac surgical outcomes. Thus, we have no data on risk factors or outcomes before the institution of the public reporting program, which limits our ability to comment upon comparisons of Massachusetts and national data before 2002.
During the study period, STS practice was to regard postdischarge patients with unknown or missing 30-day vital status as alive, which could potentially introduce bias. However, in-hospital deaths are recorded with near 100% completeness and these represent the vast majority of 30-day deaths. STS simulations (available upon request) suggest that any errors introduced by the false assumption that a patient with missing or unknown 30-day status is alive have negligible influence on hospital mortality results compared with random sampling error. Also, given 100% complete 30-day vital status for Massachusetts patients, if any STS national patient were misclassified as alive, correcting this would only increase the apparent mortality advantage for Massachusetts.
We do not have data on patients who may have not been offered surgery because of their high risk, which limits our assessment of risk aversion.
Conclusions
Mandatory Massachusetts public reporting did not disproportionately limit overall access to cardiac surgery services, as demonstrated by overall CABG volume trends from 2002 to 2014. Moreover, using comparable data and risk models, CABG outcomes in Massachusetts from 2003 to 2014 were consistently and significantly superior to national [STS] results. However, because extensive longitudinal data are unavailable before the initiation of Massachusetts public reporting, we cannot assume a causal association of report cards and superior outcomes. Finally, evidence for risk aversion was inconsistent and inconclusive. These findings, together with previous studies from New York and from STS, support the continued study of public reporting as 1 mechanism to assist patients in selecting providers and to promote optimal surgical outcomes.
Our experience suggests that surgeons in public reporting states may benefit from the establishment of formal peer organizations (such as the MA STS) that can serve as their interface with state regulatory agencies. This allows them to be part of the measure development process, to provide clinical context for problematic statistical results, and to offer remedial recommendations for outlier programs. Robust surgeon involvement in the public reporting process was also evident in the Massachusetts peer adjudication process for risk factor coding. This provides valuable oversight and education regarding appropriate coding practices, reassures surgeons and hospitals regarding data integrity, and may mitigate risk aversion by fostering provider trust in the system. It is most easily operationalized at the state or regional levels.
Finally, the Massachusetts experience demonstrates that if outlier hospitals implement appropriate programmatic, structural, leadership, and staff changes, the negative consequences of their outlier status may be temporary and remediable.
Drs Shahian, Torchiana, D'Agostino, Engelman, Rawn, and Birjiniuk have been unpaid, volunteer leaders in the development and implementation of the Massachusetts public reporting system. Dr Normand has been a paid leader for and Ms Lovett and Mr Cioffi are employees of Mass-DAC, the entity that administered the Massachusetts cardiac surgery public reporting program during the term of this study. Dr Habib is director of the Society of Thoracic Surgeons Research Center. All other authors have nothing to disclose with regard to commercial support.
The authors thank the Massachusetts Department of Public Health for the use of their data, and the Society of Thoracic Surgeons for providing data from its Adult Cardiac Surgery Database.
Appendix
Figure E1Risk prevalence plots for age and ejection fraction. For these 4 risk factors, MA [STS] (MA data submitted directly to STS) and Mass-DAC prevalence rates were virtually superimposable. These rates were generally higher than national STS rates (excluding MA patients). MA and national STS temporal trends in these risk factors were similar except for a sharp decrease in EF < 0.30 prevalences in MA patients from 2003-2006. STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
Figure E2Risk prevalence plots for age and comorbid conditions. For these 4 risk factors, MA [STS] (MA data submitted directly to STS) and Mass-DAC prevalence rates were nearly identical. These rates and their temporal trends were generally similar to those of national STS data (excluding MA patients). STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
Figure E3Risk prevalence plots for sex and comorbid conditions. For these 4 risk factors, MA [STS] (data submitted directly to STS) and Mass-DAC prevalence data were nearly identical. Massachusetts prevalence rates were consistently less than national STS rates (excluding MA). Temporal trends were similar between Massachusetts and national data for female sex, reoperation, and morbid obesity, but divergent for chronic lung disease (national STS slightly increasing, Massachusetts stable to slightly decreasing). STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
Figure E4Episodic peer adjudication. Peripheral vascular disease and preoperative dialysis underwent additional peer-adjudication only for a few years (circled in the plots). During the period of adjudication, peripheral vascular disease prevalences were lower in Mass-DAC data (additional peer adjudication) than in unadjudicated MA [STS] data or national [STS] data (excluding MA). Preoperative dialysis prevalence rates were virtually identical in adjudicated Mass-DAC data and MA data submitted directly to STS (MA[STS]), and these rates were lower than national [STS] prevalence rates. STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
Figure E5Routinely peer-adjudicated variables. For all three routinely peer-adjudicated risk factors, Massachusetts Data Analysis Center (Mass-DAC) prevalence rates were lower than MA [STS] rates (submitted directly to STS) in the early years of MA public reporting. With increasing experience and education of data managers and surgeons, the Mass-DAC and MA[STS] rates converged (vertical dotted line) for all 3 risk factors but remained substantially lower than national STS rates (excluding MA). MI, Myocardial infarction; STS, Society of Thoracic Surgeons; MA, Massachusetts; Mass-DAC, Massachusetts Data Analysis Center.
Figure E6Massachusetts and national Society of Thoracic Surgeons (STS) 30-day observed mortality rates, 2003 to 2014. In 2003, observed 30-day mortality rates were similar for MA CABG patients (whether data were submitted directly to STS (MA [STS]) or analyzed by Mass-DAC) compared with national STS mortality rates (excluding MA). MA rates then decreased substantially and were consistently lower than national STS rates thereafter, although they increased after 2011. MA [STS], Massachusetts data submitted directly to STS; Mass-DAC, Massachusetts Data Analysis Center.
Figure E7Ratios of observed to expected operative mortality for CABG, Massachusetts versus national STS results. For each public reporting year, using the same STS 2008 risk models, the ratios of observed to expected (O/E) operative mortality for MA patients in the STS Database were consistently less than one and also lower than national STS O/E ratios (excluding MA patients).
Figure E8Massachusetts Data Analysis Center (Mass-DAC) 30-day observed and expected CABG mortality rates. Because Mass-DAC expected mortality rates were derived from risk models based only on the 14 MA programs and were re-estimated annually, observed and expected rates tracked closely.
Table E1Vital status ascertainment or verification
External resources used include:
•
Acute hospital case mix data from the Massachusetts Center for Health Information and Analysis Acute Hospital Case Mix Database (http://www.chiamass.gov/case-mix-data/).
Massachusetts Data Analysis Center percutaneous coronary intervention registry data to validate prior percutaneous coronary intervention and other patient clinical data.
Table E2Isolated coronary artery bypass grafting (CABG) volumes: Society of Thoracic Surgeons (STS) national (without Massachusetts [MA]) per program and STS national (without MA) for 388 continuously participating STS programs, 2002 to 2014; New York decrease 2002 to 2014
Data from 388 STS Adult Cardiac Surgery Database sites that reported cases for all years between 2002 and 2014. New York CABG volumesNew York: 16,120 isolated CABG in 2002, 7942 in 2014 = 50.7% decrease. New York Data Sources from: 1. 2004 Adult cardiac surgery in New York State 2000-2002. Available at: https://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2000-2002_adult_cardiac_surgery.pdf. Accessed September 29, 2017. 2. 2017 Adult cardiac surgery in New York State 2012-2014. Available at: https://www.health.ny.gov/statistics/diseases/cardiovascular/heart_disease/docs/2012-2014_adult_cardiac_surgery.pdf. Accessed September 29, 2017.
No. of sites
Isolated CABG (n)
No. of sites
Isolated CABG (n)
2002
495
151,799
388
124,037
2003
539
150,801
388
116,379
2004
607
147,884
388
105,409
2005
704
150,616
388
94,890
2006
787
158,046
388
89,513
2007
859
161,167
388
84,549
2008
933
165,055
388
80,115
2009
965
164,228
388
75,593
2010
996
157,838
388
71,251
2011
1016
144,718
388
64,189
2012
1026
138,408
388
60,094
2013
1038
138,771
388
60,146
2014
1041
137,702
388
59,104
2015
1045
141,039
388
59,369
2016
1046
140,853
388
58,672
Decrease per site STS overall = (151,799/495-140,853/1046)/306.7 = 56.1%. Decrease at 388 sites with continuous STS participation 2002 to 2014 = (124,037-58,672)/124,037 = 52.7%. CABG, Coronary artery bypass grafting.
∗ Data from all sites contributing to STS Adult Cardiac Surgery Database other than Massachusetts (out of 1253 possible sites).
The New York State cardiac registries: history, contributions, limitations, and lessons for future efforts to assess and publicly report healthcare outcomes.
Impact of independent data adjudication on hospital-specific estimates of risk-adjusted mortality following percutaneous coronary interventions in massachusetts.
Contemporary impact of state certificate-of-need regulations for cardiac surgery: an analysis using the Society of Thoracic Surgeons' National Cardiac Surgery Database.
Association of public reporting for percutaneous coronary intervention with utilization and outcomes among Medicare beneficiaries with acute myocardial infarction.
Quality of care information makes a difference: an analysis of market share and price changes after publication of the New York State Cardiac Surgery Mortality Reports.
Mandatory public reporting has been generally unpopular since it began in New York in 1991, and it remains controversial today. Opponents emphasize a variety of purported evils (gaming, shaming, risk-aversion, meaningless distinctions, etc). Shahian and colleagues1 show that outcomes for isolated coronary artery bypass grafting procedures in Massachusetts improved between 2003 and 2014 and were significantly better than the outcomes reflected in the Society of Thoracic Surgeons database over the same period.
33516-5&id=fx1.jpg){kind=link}
33516-5&id=gr1.jpg){kind=link}
33516-5&id=gr2.jpg){kind=link}
33516-5&id=gr3.jpg){kind=link}
33516-5&id=gr4.jpg){kind=link}
33516-5&id=gr5.jpg){kind=link}
33516-5&id=gr6.jpg){kind=link}
33516-5&id=fx2.jpg){kind=link}
33516-5&id=fx3.jpg){kind=link}
33516-5&id=fx4.jpg){kind=link}
33516-5&id=fx5.jpg){kind=link}
33516-5&id=fx6.jpg){kind=link}
33516-5&id=fx7.jpg){kind=link}
33516-5&id=fx8.jpg){kind=link}
33516-5&id=fx9.jpg){kind=link}
33516-5&id=fx10.jpg){kind=link}