Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers:  The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS) FREE

The US Army suicide rate, although historically below the civilian rate, has increased since 2004¹ to exceed the civilian rate.² Despite numerous efforts to address this problem, including universal interventions (eg, Ask/Care/Escort prevention education and depression, posttraumatic stress disorder, and suicide screening in all primary care encounters) and high-risk interventions (eg, postdeployment screening),³ the US Army suicide rate has continued to increase. One potentially important group for targeted interventions is soldiers recently discharged from inpatient psychiatric treatment. Such patients have long been known to have a high risk of suicide.⁴ US military administrative data document an 8-fold elevated suicide risk in the 3 months after psychiatric hospitalization and a 5-fold elevated risk for the remainder of the 12 months after hospitalization.⁵ A report⁶ on the similar patterns among civilians called for expansion of posthospitalization suicide preventive interventions, noting that such interventions in the United Kingdom (eg, required outpatient visits within 1 week of hospital discharge, assertive outreach for missed outpatient appointments, 24-hour community crisis teams, and intensive community support for patients difficult to engage in traditional services) were associated with significant before-after reductions in posthospitalization suicides.⁷

Suicide is a rare outcome even among recently discharged psychiatric inpatients⁸; therefore, the benefits of providing intensive posthospitalization suicide prevention interventions to all recently discharged inpatients are low. A more rational allocation of treatment resources would be to combine relatively inexpensive universal interventions⁹ with more intensively targeted high-risk interventions.⁴ However, this tiered approach would require developing a reliable risk stratification scheme. The US Department of Veterans Affairs (VA) and the US Department of Defense (DoD) called for this kind of differentiation in their Clinical Practice Guideline (CPG) entitled Assessment and Management of Patients at Risk for Suicide.¹⁰ However, the CPG provided little concrete guidance on how these assessments should be implemented. Research has consistently revealed that health care professionals are not accurate in making such assessments.^11- 14

One potentially promising approach to assessing posthospitalization suicide risk would be to use administrative data available during hospitalization to generate an actuarial posthospitalization suicide risk algorithm. Previous research has revealed that actuarial suicide prediction is much more accurate than prediction based on clinical judgment.^11- 14 An increasing number of computerized risk algorithms are being used as clinical decision support tools in other areas of medicine and have been found to improve clinical processes.^15,16 Skepticism exists about developing such an algorithm for posthospitalization suicide interventions based on the relatively weak associations found in previous research¹⁷ on in-hospital predictors and subsequent suicides. However, a stronger risk algorithm might be developed in the US Army because of the availability of integrated administrative data for all US Army personnel. Absence of such data in the general population is widely recognized as an impediment to big data health care solutions.¹⁸ A number of empirical studies^19- 23 have documented strong predictive associations between integrated US Army and DoD administrative data and subsequent US Army suicides, although none attempted to develop a risk algorithm for posthospitalization suicides. The objective of this study was to develop such an algorithm using administrative data from the Historical Administrative Data System (HADS) of the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS).²⁴

Creation and analysis of the consolidated and deidentified data system were approved by the Human Subjects Committees of the Uniformed Services University of the Health Sciences for the Henry M. Jackson Foundation (the primary grantee), the University of Michigan Institute for Social Research (site of the Army STARRS Data Enclave), and Harvard Medical School (site of data analysis). Obtaining informed consent from individual soldiers, most of whom were no longer in service at the time the HADS was constructed, was not required because the data were deidentified.

There were 53 769 regular US Army hospitalizations from January 1, 2004, through December 31, 2009, with any International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) psychiatric admission diagnosis exclusive of tobacco use disorders (eTable 1 at http://www.armystarrs.org/publications). These hospitalizations involved 40 820 soldiers (30 763 with 1 hospitalization, 6929 with 2, and 3128 with >2), representing 0.9% of all regular US Army soldiers in any 12-month period. We excluded the 13 936 additional hospitalizations in which nicotine dependence was the only psychiatric diagnosis because these were invariably for physical disorders and nicotine dependence was noted based on withdrawal during hospitalization. There was no elevated posthospitalization suicide risk among these soldiers. We also excluded the 406 additional hospitalizations that occurred through emergency departments because of a suicide attempt without an accompanying ICD-9-CM psychiatric diagnosis. Four of these 406 soldiers died in the hospital, whereas none of the others died by suicide in the next 12 months. On the basis of evidence from another study²⁵ indicating that predictors of posthospitalization suicide vary with time since discharge and elevated risk persists 12 months after discharge, a discrete-time person-month survival file was created to examine suicides in the 12 months after hospital discharge, censoring all person-months at the beginning of new hospitalizations or terminations of active duty and allowing interactions between substantive predictors and time since hospital discharge. All person-months with suicide were coded 1 on the outcome, and all others were coded 0. This file contained 334 936 person-months for a mean of 6.2 months (334 936 per 53 760 months) after hospital discharge. This low mean reflects high rates of termination of service and subsequent hospitalization within 12 months of each hospitalization.

The HADS includes data from 38 US Army and DoD administrative data systems²⁶ (eTable 2 at http://www.armystarrs.org/publications). In a comprehensive review of published studies of predictors of civilian posthospitalization suicides, Troister et al²⁷ found 5 replicated classes of predictors: (1) sociodemographics (the most consistent being male sex and recent job loss), (2) history of prior suicidal behaviors, (3) quality of care (eg, low continuity of care), (4) time since hospital discharge (inversely related to suicide risk), and (5) other psychopathological risk factors (the most consistent being nonaffective psychosis, mood disorders, and multiple comorbid psychiatric disorders). Other studies^17,28,29 found similar predictors. We extracted HADS variables operationalizing these predictors and added US Army career variables found to predict military suicides,^19- 22 unit variables, criminal justice variables (violent crime victimization or perpetration), and measures of registered weapons. All predictors other than those that involved the hospitalization were defined as of the month before hospitalization, whereas predicted suicides were in the 12 months after hospital discharge.

We cast a wide net in extracting HADS measures of the predictor constructs. For example, we distinguished 23 categories of psychiatric diagnoses defined largely by aggregated ICD-9-CM codes (eg, attention-deficit/hyperactivity learning disorders [ICD-9-CM codes 314.0-315.9]), 8 additional categories of behavioral stressors (eg, marital problems, other stressors or adversities, suicidal ideation, and self-damaging behavior), and summary measures of any prior admission diagnoses, admission count variables, and parallel outpatient variables (eTable 1 at http://www.armystarrs.org/publications). We also included National Drug Code psychotropic medication codes collapsed into 15 categories (eg, antianxiety, antidepressant, and antipsychotic) and 25 subcategories (eg, selective serotonin reuptake inhibitor, serotonin-norepinephrine reuptake inhibitor, and tricyclic antidepressant) based on the First Databank Enhanced Therapeutic Classification System (http://www.fdbhealth.com) (eTable 3 at http://www.armystarrs.org/publications). A total of 421 individual variables were constructed (eTable 4 at http://www.armystarrs.org/publications).

Because the HADS data systems were not developed for research, more data were missing and inconsistent in some (eg, sociodemographic) component data sets than in research data sets. However, because the HADS data sets are updated monthly, missing values typically appeared in earlier and/or later months, allowing nearest neighbor imputations. Remaining missing values were resolved using randomly selected multiple imputations.³⁰ Inconsistencies were reconciled using rational imputations (eg, a soldier classified female one month but male other months was recoded male).

Discrete-time (person-month) survival analysis³¹ was used to predict suicides in the 12 months after hospitalization in 3 steps. First, functional forms of bivariate associations were examined and predictors transformed (usually sets of nested dichotomies but some collapsed-truncated continuous variables) to explore nonlinear multivariate associations. Second, all predictors were discretized and analyzed with 100 regression trees in distinct bootstrap pseudo-samples using the R package rpart program³² to prevent overfitting³³ and allow detecting interactions among predictors.^25,28 Third, predictors having significant bivariate associations and interactions emerging in 10% or more of regression trees were included as predictors in multivariate survival models.

A central challenge in the third step was multicollinearity among the 421 predictors. The classic way to address this problem is with stepwise analysis,³⁴ but this approach overfits.³⁵ Machine learning methods reduce overfitting.^36,37 The machine learning method we used was the elastic net,³⁸ a penalized regression method that provides stable and sparse estimates of model parameters by explicitly penalizing overfitting with a composite penalty λ{MPP × P_lasso + (1 − MPP) × P_ridge}, where MPP is a mixing parameter penalty with values between 0 and 1 that controls relative weighting between 2 types of penalties: the lasso penalty and the ridge penalty. The parameter λ controls the total amount of penalization.³⁹ The ridge penalty handles multicollinearity by shrinking all coefficients smoothly toward 0 but retains all variables in the model.⁴⁰ The lasso penalty allows simultaneous coefficient shrinkage and variable selection, tending to select at most one predictor in each strongly correlated set but at the expense of giving unstable estimates in the presence of high multicollinearity.⁴¹ The elastic net approach of combining the ridge and lasso penalties has the advantage of yielding more stable and accurate estimates than either the ridge or lasso alone while maintaining model parsimony.³⁸

The 3-step approach of combining regression trees with penalized regression for variable selection enabled us to incorporate possible interactions and nonlinearities in a clinically meaningful way while controlling for possible overfitting. The R package glmnet program⁴² was used to estimate penalized models with MPPs of 0.1, 0.4, 0.7, and 1.0 (an MPP of 0.0 was not used because of multicollinearity in the full predictor set). Internal 10-fold cross-validation selected the coefficient in front of the penalty. Comparative fit across the 20 specifications (ie, 4 MPP values for each of 5 constraints on the number of predictors) was evaluated by inspecting the area under the receiver operating characteristic curve (AUC) and concentration of risk (CR). The CR is the proportion of observed suicides after hospitalizations in each ventile (ie, 20 groups of hospitalizations of equal frequency) ordered by predicted suicide risk. Suicide risk of each hospitalization was calculated using coefficients to project risk as of 12 months after hospital discharge regardless of observed hospitalization data and censoring and standardized by time of hospitalization to adjust for temporal variation in suicide risk. Given that the number of hospitalizations per ventile was much larger than the number of suicides, we focused on the CR in the highest-risk ventile in selecting the best penalized model.

Once a best penalized model was selected, a conventional discrete-time survival model with a logistic link function was estimated using the same predictors as the best penalized model to examine how much the penalty reduced model fit. Because the variance inflation factor of coefficients in this model revealed estimates to be unstable, we also used forward stepwise analysis with a .05-level entry criterion to select a stable subset of predictors for a reduced version of the logistic model. Coefficients in this reduced logistic model were then exponentiated to create odds ratios (ORs) for ease of interpretation. Ventiles from the best penalized model were then collapsed into risk strata using the logic of stratum-specific likelihood ratios.⁴³ The CR, AUC, and the standardized (for amount of uncensored time observed after each hospitalization) suicide rates per 100 000 person-years were calculated for these risk strata. Finally, parallel rates of risk were calculated for unintentional injury deaths, attempted suicides, and subsequent hospitalizations in the same ventiles to evaluate other adverse outcomes associated with posthospitalization suicide risk.

Sixty-eight hospitalized soldiers died by suicide within 12 months of hospital discharge (263.9 suicides per 100 000 person-years vs 18.5 suicides per 100 000 in the total US Army),²³ representing 12.0% of all US Army suicides. An additional 157 hospitalized soldiers died in other ways, and 22 010 others terminated active duty for other reasons (eg, administrative separation and retirement) within 12 months of hospital discharge.

No interactions emerged in more than 10% of regression trees. However, 131 of the 421 bivariate associations (31.1%) between individual predictors and suicides were significant at the .05 level (eTables 5-9 and eTables 11-15 at http://www.armystarrs.org/publications). All these variables were used in the penalized multivariate models.

A 10-fold cross-validation revealed that AUC was maximized across the 20 penalized survival models for an MPP of 1.0 (lasso) with 73 predictors and an MPP of 0.1 to 0.7 with 72 to 122 predictors (Figure 1). Because the lasso model yielded the best cross-validated CR in the highest-risk ventile (52.9%) (Table 1), we estimated a conventional discrete-time survival model with a logistic link function using the same 73 predictors. This model had a much higher AUC (AUC, 0.89) and CR (CR, 61.8%) in the highest-risk ventile than the lasso model with the same predictors, but this was because of overfitting (variance inflation factor >5 for 6 coefficients). Forward stepwise analysis selected a more stable set of predictors in a reduced logistic model, and this model, which contained 20 predictors, had a slightly lower AUC (AUC, 0.84) and CR (CR, 50.0%) in the highest-risk ventile than the lasso model.

Caution is needed in interpreting predictors in the reduced logistic model because the variable selection algorithm maximized overall prediction accuracy rather than individual coefficient accuracy. It is nonetheless noteworthy that the model included variables in all predictor classes (Table 2): 3 sociodemographic characteristics (male sex, enlistment at ≥27 years of age, and US Armed Forces Qualification Test score >50th percentile; ORs, 1.9 [95% CI, 1.0-3.5] to 7.9 [95% CI, 1.9-32.6]), access to firearms (number of registered pistols; OR, 1.3; 95% CI, 1.0-1.6), crime perpetration (weapons possession or verbal assault; ORs, 2.2 [95% CI, 1.2-4.0] to 5.6 [95% CI, 1.7-18.3]), prior suicidality (ORs, 1.6 [95% CI, 1.1-2.5] to 2.9 [95% CI, 1.7-4.9]), prior psychiatric treatment (ORs, 0.3 [95% CI, 0.2-0.6] to 5.6 [95% CI, 1.8-17.7]), and characteristics of the focal hospitalization (ORs, 0.4 [95% CI, 0.2-0.7] to 6.0 [95% CI, 2.1-17.4]). The 2 ORs less than 1.0 were for (1) being above the 50th percentile on the ratio of number of psychiatric hospitalizations to time in service and (2) posttraumatic stress disorder during current hospitalization.

Inspection of the CR across predicted risk ventiles led to creation of 4 risk strata. Most suicides occurred in the highest-risk stratum (which was made up of the 5% of hospitalizations in the highest-risk ventile; CR, 52.9%) (Figure 2). The CR was lower (CR, 8.8%) in the second stratum (made up of the 5% of hospitalizations in the second-highest ventile), lower still (CR, 4.2%) in a third stratum (made up of the 35% of hospitalizations in the next 7 ventiles), and lowest (CR, 0.8%) in the fourth stratum (made up of the 55% of suicides in the lowest 11 ventiles).

Suicide risk ranged from 1338.8 per 100 000 hospitalizations in the highest-risk stratum to 20.3 per 100 000 hospitalizations in the lowest-risk stratum (Table 3). However, because mean time in service after hospital discharge was considerably less than 12 months, suicide risk per 100 000 person-years was considerably higher than per 100 000 hospitalizations: 3824.1 per 100 000 person-years in the highest-risk stratum to 40.9 per 100 000 in the lowest-risk stratum.

The CR in the highest-risk stratum did not differ significantly, depending on whether (1) hospitalization was in a facility with a mental health inpatient unit vs a general medical facility without such a unit (48.2% vs 66.7; χ²₁ = 1.7; P = .19); (2) the suicide occurred before vs after September 1, 2008 (median date of suicides during the study period; 38.7% vs 70.3%; χ²₁ = 2.4; P = .12); or (3) the suicide did vs did not occur within 3 months of hospital discharge (median time to postdischarge suicide; 52.6% vs 56.7%; χ²₁ = 0.0; P = .99).

Soldiers in the highest-risk stratum also had elevated risks of other adverse outcomes in the year after hospital discharge, including unintentional injury deaths (CR, 10.1%; χ²₁ = 7.1; P = .008), suicide attempts (CR, 9.1%; χ²₁ = 332.7; P < .001), and subsequent hospitalizations (7.5%; χ²₁ = 893.4; P < .001). Soldiers in the highest predicted suicide risk stratum had 7 unintentional injury deaths, 830 suicide attempts, and 3765 subsequent hospitalizations within 12 months of hospital discharge (492,666.2 per 100 000 person-years). At least one of these outcomes occurred after 46.3% of the highest-risk hospitalizations.

Although risk factors for suicide are widely known, synthesizing this information to optimize suicide prediction has been an elusive goal up to now. This study addressed this problem by using machine learning to generate an actuarial suicide risk algorithm from US Army and DoD administrative data, finding that 52.9% of suicides occurred after the 5% of hospitalizations with highest predicted risk. Although interventions in this high-risk stratum would not solve the entire US Army suicide problem given that posthospitalization suicides account for only 12% of all US Army suicides, the algorithm would presumably help target preventive interventions. Before clinical implementation, though, several key issues must be addressed.

The first question is whether the risk algorithm is sufficiently stable to predict future suicides given that it is based on only 68 prior suicides. It is noteworthy that the machine learning methods used to create the algorithm were designed explicitly to maximize stability of predictions. Within-sample stability analyses found that the CR did not vary significantly by type of inpatient facility, year of hospitalization, or number of months since hospital discharge; however, this does not guarantee future stability. Algorithm stability will consequently be tested again in the 2010-2013 US Army suicide data in a future study to address this question.

The second question is whether the risk algorithm improves on clinical judgment. The study was unable to examine this issue empirically because the US Army electronic medical record does not include a structured field where health care professionals must record suicide risk assessments. In addition, documentation of suicide risk assessment in clinical notes was not consistent during the study period. However, with improved documentation after the VA and DoD CPG, comparison of actuarial to clinical prediction may be possible in the future. As noted in the Introduction, though, previous research has indicated that actuarial suicide prediction is much more accurate than prediction based on clinical judgment.^11- 14 This evidence is consistent with a large body of literature reporting that actuarial methods are superior to expert judgments in many areas of prediction.^45,46 At the same time, the comprehensive suicide risk assessments required by the new VA and DoD CPG¹⁰ will generate information not included in administrative records. As a result, our algorithm should be seen as a component of this comprehensive clinical assessment rather than a substitute for this assessment.

The third question is whether suicide is sufficiently common in the highest-risk stratum and available interventions sufficiently powerful to make targeted posthospitalization interventions efficient compared with alternative ways of deploying the same clinical resources. Our results shed no light on this question. The potential for harm also has to be taken into consideration because intensive posthospitalization interventions might lead to undue scrutiny by nonmedical leaders that adversely affect soldier careers. This concern is all the more important given that most soldiers identified as being high risk do not commit suicide. Although a formal analysis of comparative risks and benefits is beyond the scope of this report, it is noteworthy that the highest-risk stratum had significantly elevated risks of other adverse outcomes and that prevalence of at least one such outcome was present after 46.3% of highest-risk hospitalizations. Ameliorative effects of expanded high-risk interventions on these outcomes (ie, unintentional injury deaths, suicide attempts, and subsequent hospitalizations) are plausible because numerous risk factors for suicide (eg, depression and substance abuse) are also risk factors for these other outcomes^2,47,48 and most suicide prevention interventions recommended for high-risk patients are likely to affect these outcomes as well.^7,10 These presumed benefits would have to be considered in a broad-based evaluation of risks and benefits of any future targeted high-risk posthospitalization preventive interventions.

The major limitations of our analysis involve errors in the administrative data used as predictors (missing and inconsistent values and errors in ICD-9-CM diagnoses). In addition, the algorithm could almost certainly be improved if more nuanced risk factor data were available. Because the new VA and DoD CPG contains a checklist of risk factors health care professionals are urged to assess in evaluating suicide risk, creation of a system to record these assessments in the electronic medical record along with the health care professional’s clinical global impression of patient suicide risk might increase the completeness of these assessments and provide a rich source of information for future risk algorithm refinement.

The high concentration of risk of suicides and other adverse outcomes might justify targeting expanded posthospitalization interventions to soldiers classified as having highest posthospitalization suicide risk, although final determination requires careful consideration of intervention costs, comparative effectiveness, and possible adverse effects.