External Validation of the UKPDS Risk Engine in Incident Type 2 Diabetes: A Need for New Type 2 Diabetes–Specific Risk Equations

National policies for the management of both cardiovascular disease (CVD) and type 2 diabetes advocate the calculation of CVD risk in order to identify high-risk patients for targeted interventions (1–5). Several multivariable risk prediction models (or risk scores) have been developed for the general, nondiabetic population that also account for diabetes, but only a few are specific to type 2 diabetes (6). Only two of these have been developed in patients with newly diagnosed type 2 diabetes, and they both use data from the UK Prospective Diabetes Study (UKPDS) (7). These two models, one for coronary heart disease (CHD) and the other for stroke, combine to form the UKPDS Risk Engine (UKPDS-RE) (8,9).

International and national clinical guidelines recommend using the UKPDS-RE for predicting cardiovascular risk (5,10–12). Not only is the UKPDS-RE advocated for communicating cardiovascular risk to diabetic patients (13), it has been relied upon for public health decisions (14–17). Evidence that these equations are inadequate could bring into question the evidence base underpinning many clinical decisions and public policies about the management of type 2 diabetes. Two systematic reviews of external validations of type 2 diabetes cardiovascular risk prediction models (6,18) reported poor calibration of the UKPDS-RE CHD equations in 10 separate studies (19–28) and differing findings for the stroke equations in two separate studies (23,25). The largest of these studies from the U.K. used only a small sample (n = 798) from a single locality (21). The largest international study had a larger but still relatively small sample size (n = 7,502) using data collated from 20 countries (23).

The purpose of this study was to carry out an external evaluation of the performance of the UKDPS-RE on a large, relatively contemporary dataset of U.K.-resident patients newly diagnosed with type 2 diabetes.

This study was performed using data from the Clinical Practice Research Datalink (CPRD) and linked data from the Office for National Statistics (ONS) and Hospital Episode Statistics (HES). Ethical approval for the study was granted by the CPRD Independent Scientific Advisory Committee on 6 September 2012, protocol number 12_084R.

The CPRD observational dataset consists of longitudinal, anonymous records from nearly 700 primary care practices and >11 million patients throughout the U.K. (based on the January 2012 release) (29). The computerized data, recorded in the course of routine health care by general practitioners (GPs) and associated staff, include demographic and lifestyle information, medical history, clinical investigations, drug prescriptions, and hospital referrals. Diagnoses in CPRD are recorded using the Read code classification and have been validated in a number of studies, showing a high positive predictive value (30).

Additionally, 357 of the English practices contributing to the dataset, representing ∼45% of CPRD patients, participate in a linkage scheme by which registered patients are anonymously linked, through a trusted third party, to other independent datasets (31). These include hospital admission data, collated nationally for England as the HES (32), and mortality data, collated by the ONS (33). HES provides details of all National Health Service (NHS) inpatient admissions in England since 1997, including primary and contributory causes coded using the ICD-10 classification. ONS provides details of all deaths in England, with immediate and antecedent causes coded using the ICD-9 and ICD-10 classifications.

For this study, a single cohort of patients with incident type 2 diabetes, registered with practices between 1998 and 2011, was identified from the CPRD dataset as described below. In order to improve ascertainment of cardiovascular events, only patients whose records linked to the HES and ONS mortality datasets were included, with the former providing details of diagnoses and procedures related to inpatient episodes, and the latter providing both the date and cause(s) of death. The HES data also provided the ethnicity information required for the study. Patients aged between 35 and 84 years at diagnosis were included in the study. As the original UKPDS-RE was based on a cohort aged <65 years, a sensitivity analysis was performed. Patients were excluded if they had ongoing or recent CVD (as defined by the UKPDS study criteria), implausible or improbable dates, or missing or indeterminate sex or smoking status. Patients that were HES-eligible but had no records in the linked HES data were excluded (n = 1,727). Patients whose ethnicity was not recorded (n = 29,199) were presumed Caucasian and combined with the Caucasian group.

Patients were considered for selection if they had a clinical (Read or ICD-10) code indicative of diabetes in their CPRD or linked HES records. As not all clinical codes for diabetes distinguish between type 1 and type 2 diabetes, and some patient histories may have erroneously contained both type 1 and type 2 diabetes codes, these patients were categorized as having type 2 diabetes if they met one or more of the following criteria: 1) clinical codes exclusively indicative of type 2 diabetes; 2) at least one clinical code indicative of type 2 diabetes (regardless of others indicative of type 1 or nonspecific diabetes) and at least one prescription for an oral hypoglycemic agent (OHA); 3) prescription of two or more classes of OHA; and 4) diagnoses of both type 1 and type 2 diabetes and an age of diagnosis older than 35 years. Any patient with evidence of diabetes secondary to other causes was excluded.

The date of diabetes incidence was defined as the date of either first diagnosis or first prescription of a diabetes medication, whichever was earlier. A “wash-in” period of 365 days was applied to exclude nonincident type 2 diabetes cases.

The primary outcomes comprised the four cardiovascular events evaluated by the UKPDS-RE: CHD, fatal CHD, stroke, and fatal stroke. To aid comparison, the definitions of the outcomes in the CPRD cohort were the same as the definitions from the UKPDS (7–9). CHD was defined as the occurrence of fatal or nonfatal myocardial infarction (MI) or sudden death (7). In patients with multiple CHD events, only the first event was considered. No distinction was made between ischemic and hemorrhagic strokes. In patients with multiple strokes, only the first stroke was considered. Deaths from causes other than the defined outcomes of interest were treated as censored. Occurrence of clinical events of interest in CPRD was observed from GP-recorded diagnoses, diagnoses recorded during a hospital admission, or cause of death.

Values for the input variables required for the UKPDS-RE were taken from CPRD observations around the time of diabetes incidence. Table 1 shows the baseline characteristics at the time of incident diabetes. Baseline smoking status was the value recorded closest to diabetes incidence, preferring values recorded prior to diabetes incidence; for systolic blood pressure (SBP), glycated hemoglobin A_1c (HbA_1c), total cholesterol, and HDL cholesterol, the baseline value was the average of biochemical readings recorded in the first 2 years. The numbers of readings used in deriving these 2-year averages were also recorded for use as input parameters (regression dilution) in the UKPDS-RE (9). Atrial fibrillation was deemed present at baseline if a prior diagnosis or record of a CHADS₂ test existed (CHADS₂: congestive heart failure, hypertension, age ≥75 years, diabetes, prior stroke or transient ischemic attack).

Multiple imputation was used to replace missing values for SBP, HbA_1c, total cholesterol, HDL cholesterol, and the number of biochemical readings used in their 2-year averages. Multiple imputation is a technique that offers substantial improvements over value replacement approaches based on complete cases or cases matched for age and sex (34). It involves creating multiple copies of the data and imputing the missing values with plausible values randomly selected from their predicted distribution. Here, we used the multivariate imputation by chained equations (MICE) library in the R (35) statistical programming language to generate five imputed datasets. Rubin’s rules were then used to combine the results from analyses on each of the imputed values, producing estimates and confidence intervals that incorporate the uncertainty of imputed values.

For each of the four outcomes, the 10-year estimated risk was calculated for every patient in the CPRD cohort using the UKPDS-RE (8,9). Observed 10-year risks were generated using the Kaplan-Meier method, by decile of predicted risk and by 5-year age-group. The predictive performance of the UKPDS-RE on the cohort was assessed by examining measures of calibration and discrimination.

Calibration refers here to how closely the predicted 10-year cardiovascular risk agreed with the observed 10-year cardiovascular risk. This was assessed for each decile of predicted risk, ensuring 10 equally sized groups, and for each 5-year age-group by calculating the ratio of predicted to observed cardiovascular risk separately for males and females. Plotting observed proportions versus predicted probabilities, where a 45° line denoted perfect discrimination, enabled the calibration of the risk score predictions to be visually assessed.

Discrimination is the ability of the risk score to differentiate between patients who did and did not experience an event during the study period. This measure was quantified by calculating a concordance index (C index), in which a value of 0.5 represents random chance and 1 represents perfect discrimination. All statistical analyses were carried out in R (v2.15.2) (35).

We identified 79,966 eligible cases, who contributed 383,025, 388,269, 381,833, and 388,004 person-years of observed follow-up for CHD, fatal CHD, stroke, and fatal stroke, respectively. The incidence rates for cardiovascular events in the CPRD cohort were 59.2 (95% CI 56.8–61.6), 16.8 (15.5–18.1), 71.2 (68.5–73.2), and 15.2 (14.0–16.5) per 10,000 person-years for CHD, fatal CHD, stroke, and fatal stroke, respectively. The median durations of follow-up were 4.2 years (IQR 2.0–7.2), 4.3 (2.1–7.3), 4.2 (2.0–7.2), and 4.3 (2.1–7.3), respectively. The proportions of cases followed for 10 years or more were 8.5, 8.8, 8.4, and 8.8%, respectively. Table 1 details the characteristics of these patients at or in the first 2 years from diabetes diagnosis (baseline). People recruited to the UKPDS were a very unusual group of people with type 2 diabetes, and this is reflected in the baseline characteristics. For instance, the mean age at baseline for females in the UKPDS was 53 vs. 63 years in general clinical practice (Table 1).

Complete data on age, ethnicity, smoking status, atrial fibrillation status, SBP, HbA_1c, total cholesterol, and HDL cholesterol were available for 70% of females (n = 43,741) and 74% of males (n = 54,710). Most patients (n = 120,572; 88.3%) had missing data on no more than two risk factors (Supplementary Table 1). For specific covariates, the proportion of missing data were as follows: HDL cholesterol (26.2% in females and 23.6% in males), SBP (4.1% in females and 3.7% in males), HbA_1c (8.1% in females and 12.0% in males), and total cholesterol (9.5% in females and 12.3% in males) (Supplementary Table 1).

A visual illustration of the agreement between mean observed risk and the mean predicted risk, grouped by decile of predicted risk for each of the four UKPDS-RE outcomes is shown in Fig. 1. Presenting these data in an alternative way, Fig. 2 shows the agreement between the observed risk and the predicted risk by 5-year age- and sex-specific groups for each of the outcomes. Both the CHD models were clearly miscalibrated, notably for males (overestimating event rates by 174 and 466%, compared with 160 and 398% in females, for CHD and fatal CHD, respectively) and most notably for fatal CHD (overestimating event rates by 440%). There was a clear and consistent overprediction of risk across all deciles of predicted risk, and across all age- and sex-specific groups. The disagreement between observed proportions and predicted risks increased in subsequent deciles of risk and in the older age-groups (Figs. 1 and 2). The stroke model overestimated event rates by 29 and 58% in females and males, respectively, and the fatal stroke model underestimated event rates by 20% in males and overestimated these rates by 11% in females. The stroke models showed modest agreement between observed and predicted risk grouped by decile of risk, with the exception of the final, 10th decile for the stroke model in both males and females (Figs. 1 and 2). Both the stroke and the fatal stroke models showed modest agreement across all age-groups, with some divergence toward the latter age ranges (70–85 years), most noticeably for males in the stroke model. The fatal stroke model slightly underpredicted risk for the latter age-groups, whereas the stroke model tended to overpredict risk for these latter age-groups.

Table 2 summarizes the performance of the four UKPDS-RE models in predicting the 10-year risk in type 2 diabetic patients who were initially free of CVD. The UKPDS-RE overestimated the risk of CHD, fatal CHD, and stroke by 169, 440, and 44%, respectively, and underestimated the risk of fatal stroke by 5%. According to the C index, all models were found to have acceptable model discrimination, with the exception of the CHD model in males (C index = 0.65), which was found to have modest discrimination. The C-index values for females and males, respectively, were as follows: for the CHD model, 0.71 and 0.65; for the fatal CHD models, 0.78 and 0.74; for the stroke models, 0.73 and 0.71; and for the fatal stroke models, 0.77 and 0.78. All the models showed better discrimination in females, with the exception of fatal stroke, and better discrimination (and variability in estimates) in fatal outcomes in both females and males. Of all the models evaluated, fatal stroke demonstrated the best prognostic separation, with discrimination results ranging from acceptable to good (0.77 and 0.78 in females and males, respectively), whereas CHD exhibited the worst prognostic separation, most noticeably in males, with discrimination results ranging from modest to acceptable (0.71 and 0.65).

This validation study showed that the risk equations that constituted the UKPDS-RE were poorly calibrated and significantly overestimated CHD risk. The stroke equations showed calibration ranging from poor to moderate. All the UKPDS-RE equations showed moderate discrimination, with slightly better discrimination for fatal events. This finding was concordant with several other much smaller, external validation studies (<8,000 subjects) that also showed poor calibration and overestimation of CHD risk by the UKPDS-RE (19–28). To date, this is the largest study, with ∼80,000 patients, and the most comprehensive external validation of cardiovascular risk prediction in a diverse and more contemporary population with type 2 diabetes.

The relatively poor performance of the UKPDS-RE may be explained, at least in part, by the differences in the baseline profiles of the UKPDS and CPRD populations. These plausibly include the epidemiological setting, changesin life expectancy, changes in smoking habits, the presence or absence of comorbidities, temporal changes in diabetes management, and changes in the general quality of care. Other plausible explanations include the possible harm of overly aggressive treatment with sulfonylureas and insulin in the early stages of the disease (36).

The CPRD cohort used in this study was drawn from the U.K. general practice and identified 79,966 patients aged 35–85 years newly diagnosed with type 2 diabetes and registered between 1998 and 2011. The data used to derive the UKPDS-RE risk equations originated from a randomized trial of 5,102 U.K. patients aged 25–65 years newly diagnosed with type 2 diabetes and recruited between 1977 and 1991 (followed up until 1997) (7). The CPRD cohort comprised patients aged 35–85 years for two reasons: patients under 35 years of age were excluded to reduce misclassification of type 1 diabetes, and patients 66–85 years of age were included to reflect the fact that National Institute for Health and Care Excellence (NICE) guidelines recommend using the UKPDS-RE for all ages (13). Of the 79,966 patients in the CPRD cohort, 31,179 (39%) were outside the 25–65 age range, and a sensitivity analysis suggested that the inclusion of older subjects aged 65–85 years did not significantly affect calibration or discrimination.

A number of the UKPDS trial’s exclusion criteria, namely, macrovascular complications, ketonuria, nephropathy, severe retinopathy, malignant hypertension, uncorrected endocrinopathy, and severe concurrent illness, were not applied to the CPRD cohort because their presence would not preclude the use of the UKPDS-RE in clinical practice (7). It is important to note that, by the nature of trial selection criteria, UKPDS recruits were more likely to be of lower risk, suggesting that the UKPDS-RE would be expected to underestimate risk when applied to the CPRD cohort. Overall, the UKPDS-RE overestimated cardiovascular risk in the CPRD cohort, suggesting that, in spite of the additional exclusion criteria, the UKPDS patients were at higher risk.

A potential difference in the rigor of ascertainment of primary outcomes between UKPDS and CPRD warrants consideration. In this study, we deliberately limited selection to those designated by CPRD as being of research quality, with data linked to HES and ONS mortality data during their entire follow-up period. These criteria combine to make case ascertainment among the highest of any observational data sources. Even prior to the introduction of HES-linked data in CPRD, the predictive value of GP-recorded diagnoses of acute MI in the General Practice Research Database (forerunner to CPRD) exceeded 90% (37).

The secular differences between the UKPDS sample and the current CPRD cohort may have played an important role. The advent of routine diabetes screening in primary care in the U.K. has almost certainly led to earlier diagnosis of type 2 diabetes than was available at the time of UKPDS recruitment. This is supported by an absolute 2% fall in average incident HbA_1c among U.K. patients with newly diagnosed type 2 diabetes between 1991 and 2012 (38), although the mean HbA_1c at specific regimen initiation did not change at all (39). As such, patients in the UKPDS cohort are likely to have had more advanced diabetes at the point of diagnosis, with correspondingly greater vascular morbidity.

Over the same period, the diagnosis of MI has evolved from one based solely on clinical symptoms to one that may involve increasingly sophisticated serological and imaging components, such that the severity of MI on admission may plausibly have been reduced. Post-MI care has also improved over the period, and consequently death rates subsequent to MI have fallen. This may partially explain why the UKPDS-RE overestimated fatal CHD, but it does not account for the same discrepancy in nonfatal CHD, which by this rationale could be regarded as conservative.

Another explanation for the disagreement in the observed and predicted risk estimates may be the progressive increase in the use of effective medication for hypertension and dyslipidemia over the past 20–30 years. Of the CPRD patients at baseline (i.e., type 2 diabetes incidence), 22.4% were taking lipid-lowering medication, 49.2% were taking antihypertensive treatment, and 13.7% were taking some form of antiplatelet therapy at baseline. By contrast, the UKPDS was conducted at a time when the number of patients taking such medications was much lower. For example, of the UKPDS patients at baseline, 0.3% used lipid-lowering therapy, 12% used antihypertensive therapy, and 1.6% used more than one aspirin daily. Furthermore, during the period of follow-up, <2% of UKPDS patients took lipid-lowering therapy at any stage compared with 75.3% of the CPRD cohort (40).

Other changes are also apparent. Only 19% of the CPRD patients were current smokers at baseline, compared with 30% in the UKPDS. The high relative risk reduction in CHD afforded by statin therapy (subsequent to UKPDS) could have had the effect of reducing the amount of risk that was then potentially modifiable by other interventions such as new glucose-lowering therapies. The benefits of statin therapy are believed to extend beyond their effect on lipid profiles. This is plausible, given that UKPDS-RE considerably overestimated the risk of CHD but not that for stroke. On the other hand, the specific risk markers targeted by these drugs, such as cholesterol, blood pressure, and glucose control, are still accounted for within the UKPDS-RE, so the magnitude of the discrepancy remains difficult to explain.

The principal difference between the CHD and stroke models is the presence of HbA_1c as an input parameter in the former. The poor calibration of CHD in this study brings into question the role of glucose control in predicting macrovascular complications. In sensitivity analysis, where decile of observed HbA_1c was used as the subgroup criterion, there was no gradient in observed risk of CHD, contrary to widespread expectation (Fig. 3). If corroborated, this would have a significant impact on current clinical management guidelines for type 2 diabetes. Our findings might also suggest that, in contemporary practice, the “benefit” of glucose control (i.e., reduction in CHD risk) is being overstated and consequently is having an undue influence on the diagnosis and treatment of type 2 diabetes.

The overestimation of cardiovascular risk by the UKDPS-RE may also lead to unnecessary targeting of patients for preventative strategies. Accurate estimation of absolute risk is important not only for communicating information on prognosis to patients and practitioners but also for estimating the potential risk-benefit balance and cost effectiveness of therapy. For example, NICE guidelines for the management of type 2 diabetes recommend using the UKPDS-RE and a specified risk threshold to identify patients not considered to be at high cardiovascular risk for lipid-lowering therapy with statins. Due to the considerable overestimation of cardiovascular risk observed in this study, use of the UKPDS-RE in clinical practice may potentially lead to harmful overtreatment of patients with type 2 diabetes.

A major strength of this study was the size and representativeness of the cohort. Its limitations are the high levels of missing data for HbA_1c, total cholesterol, and HDL cholesterol. Omitting cases with missing data and performing a complete-case analysis would have potentially introduced bias into the study. However, the issue of missing data was addressed by using established methods of multiple imputation. We assumed that people with missing ethnicity data were white. This may have biased the findings to some small degree, but it is unlikely to have impacted substantially on our findings.

Measurement error in identifying the CVD outcomes was present in the analysis, but this study has endeavored to apply the UKPDS study’s definitions of the cardiovascular outcomes as far as possible in selecting appropriate medical codes (7). Moreover, we supplemented the clinical information recorded in the CPRD with linked but independent secondary care data from HES, which included details of primary and additional diagnoses for inpatient episodes, and with cause-specific mortality data extracted from death certificates from the ONS. It is therefore unlikely that measurement error is a large source of bias.

Restricting cohort membership to patients from the subset of English practices participating in the linkage scheme between CPRD and HES/ONS should not have introduced significant bias; patient characteristics have been found to be similar between linked and nonlinked practices (30). In order to provide data on ethnicity, only those HES-eligible patients with a hospital contact were included in our cohort. This excluded only 2% of patients, but these patients were presumably healthier than the overall cohort.

Here we have attempted to validate the UKPDS-RE as a prognostic tool in a cohort of newly diagnosed subjects. We did not evaluate its performance with respect to CVD risk among patients with established type 2 diabetes. As the CHD and stroke models each include duration of diabetes as an input parameter, exploration of the utility of UKPDS-RE among prevalent cases of type 2 diabetes is an important future objective.

The four UKPDS risk equations constituting the UKPDS-RE showed a reasonable ability to identify high-risk patients (discrimination) but were generally poor at quantifying the absolute risk (calibration). The UKPDS-RE CHD risk equations consistently overestimated absolute risk, whereas the UKPDS-RE stroke equations performed relatively well. However, when considered as a whole, the UKPDS-RE was unsuitable for predicting CVD risk in U.K. subjects with newly diagnosed type 2 diabetes. Our findings suggest that the use of UKPDS-RE in clinical practice will lead to overestimation of CVD risk in patients with newly diagnosed type 2 diabetes. This in turn is likely to lead to selection of preventative treatments, for which, for some patients, the balance of risks may outweigh the benefits. Considering the widespread application of these prediction models in clinical practice, drug reimbursement, and public health decision making, we suggest that there is a need for revised risk equations in type 2 diabetes.

Funding. C.A.B. has received a scholarship from the Medical Research Council (MRC) and the Engineering and Physical Sciences Research Council (EPSRC). C.J.C. has received research grants from the EPSRC, the MRC, and the NHS.

Duality of Interest. All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare no support from any organization for the submitted work. C.D.P. and C.Ll.M. have been employed and S.J.-J. is employed by Pharmatelligence, a research consultancy receiving funding from pharmaceutical companies. C.J.C. has received research grants from various health-related organizations, including Abbott, Astellas, Diabetes UK, the European Association for the Study of Diabetes, Ferring, GlaxoSmithKline, Eli Lilly and Company, Medtronic, Merck Sharp & Dohme, Pfizer, Sanofi, Shire, and Wyeth, and consults for Amylin, Aryx, Astellas, Boehringer Ingelheim, Bristol-Myers Squibb, Diabetes UK, Eisel, Ferring, GlaxoSmithKline, Ipsen, Eli Lilly and Company, Medtronic, Merck Sharp & Dohme, Pfizer, Sanofi, Takeda, and Wyeth. No other potential conflicts of interest relevant to this article were reported.

Author Contributions. C.A.B. contributed to study design, analyzed the data, interpreted the results, and drafted and revised the manuscript. C.D.P. conceived the study, contributed to study design, interpreted the results, and revised the manuscript. S.J.-J. contributed to study design, provided data preparation and technical support, and revised the manuscript. C.Ll.M. and C.J.C. contributed to study design, interpreted the results, and revised the manuscript. G.E. and I.S. interpreted the results and revised the manuscript. C.J.C. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Prior Presentation. This study was presented in abstract form at the 73rd Scientific Sessions of the American Diabetes Association, Chicago, IL, 21–25 June 2013.