Measuring the Quality of Outpatient Treatment for Schizophrenia FREE

AGENCIES THAT fund public mental health care are currently implementing management strategies that, while designed to improve access and efficiency, could have powerful unanticipated effects on the quality of care.¹ Fiscal technologies developed to manage behavioral health benefits of private corporations are being applied to public systems that care for severe, chronic disorders such as schizophrenia. Concerns that severely ill individuals may be at increased risk for poor outcomes under these new financing systems^{2 - 4} have led public agencies to increase their oversight functions.⁵ However, surprisingly little is known about how to monitor quality of care for schizophrenia,⁶ or what "benchmarks" of quality exist in treatment organizations.^{7 - 8} If we are to evaluate the impact of managed care or quality improvement efforts on severely ill populations, then we need quality evaluation methods that can be broadly implemented in public settings.⁹

Fortunately, clinical research in schizophrenia has facilitated the development of practice guidelines and national recommendations for care.^{10 - 12} A review of the literature regarding treatment of schizophrenia reveals strong evidence for the efficacy of medication management, clinical case management, and family management.¹³ Appropriate medication management reduces relapse and rehospitalization and can improve functioning and quality of life.¹⁴ Appropriate medication strategies include new approaches to old medications,¹⁵ and new, more expensive medications with improved side-effect and efficacy profiles.^{16 - 17} Both case management and family management have reduced rehospitalization rates and improved functioning.^{18 - 21}

However, these treatment recommendations have not yet been employed to evaluate quality of care for schizophrenia, and using them to do so poses challenges. Treatment recommendations are often quite complex and some may be too cumbersome for use at busy clinics. Further, it is unclear as to how much data, what kinds of data (process or outcome measures), and what sources of data (patient, medical records, or administrative data) will be needed to evaluate quality.²²

This study describes the quality of care provided to schizophrenic patients at 2 public mental health clinics. First, we defined quality care using national guidelines, and developed a simple criteria set that examines key aspects of care that are likely to have a large effect on outcomes. For medication management, criteria for effective care are well accepted and were operationalized. Since effective case management and family management are less well defined^{21 ,23 - 24} we developed criteria that focus on access to these treatments. Second, we used these criteria to evaluate treatment quality at 2 clinics. Patients were interviewed and their medical records abstracted to determine whether each patient was receiving poor-quality medication management, case management, or family management. Third, to understand the contribution of the organization and its providers to treatment quality, we examined the extent to which common quality problems could be attributed to patient as opposed to nonpatient factors. Finally, to determine whether secondary data sources can be substituted for more expensive patient-derived information, we compared quality assessment based on medical records with quality assessment based on both medical records and patient interviews.

Quality of care was examined at 2 very different publicly funded clinics. One clinic treated about 2000 outpatients as part of a large Veterans Affairs (VA) Medical Center serving a metropolitan area; and had strong academic affiliations, numerous psychiatric trainees, and a reputation for performing high-quality clinical research. The other clinic was an urban Community Mental Health Center (CMHC) funded by a regional mental health authority. It had no academic affiliations and about 600 outpatients in active treatment.

Patients were eligible for the study if they (1) had a diagnosis of schizophrenia or schizoaffective disorder, (2) were between 18 and 65 years old, (3) had been in treatment at the clinic for at least 3 months, (4) had spent no more than 21 days in the hospital during the previous 3 months, and (5) had at least 1 visit with a psychiatrist during a 3-month sampling period in early 1996. We excluded patients who had recent clinic admissions because assessment and engagement issues might take precedence over treatment change, and patients who had recent extensive hospitalizations because hospital clinicians might have determined the treatment regimen.

We used real-time, visit-based sampling to identify a random sample of adults with schizophrenia at each clinic. During a 3-month period, every patient who came to the clinics was screened using the eligibility criteria. To ensure that visit frequency did not affect the probability of being selected, patients were only screened at their first visit during the 3-month period. Subjects were then randomly selected from the population of eligible patients. Informed consent was obtained after the study had been fully explained.

Seventy percent of selected patients chose to participate. We interviewed 113 VA patients and 111 CMHC patients and abstracted their medical records. Nonresponders did not differ significantly from respondents on age, sex, or race. Subjects at the VA were 96% male, had a mean age of 49 years (SD=8.5 years), and were 47% black, 42% white, and 6% Latino. Subjects at the CMHC were 56% male, had a mean age of 44 years (SD=10.1 years), and were 88% black, 5% white, and 7% Latino. On average, VA clinic patients had been ill longer (22 vs 18 years, t=2.8, df=222, P=.005), had a higher monthly income ($1109 vs $643, t=6.8, df=211, P<.0001), and were less likely to be living with their family (38% vs 62%, χ²₁=13.0, P<.001).

Criteria for poor-quality care were based on a comprehensive review of the literature and national treatment recommendations, particularly those of the Schizophrenia Patient Outcomes Research Team (Schizophrenia PORT).^{12 ,25} We drafted criteria that were consistent with the Schizophrenia PORT recommendations, focusing on aspects of treatment that the literature review identified as most efficacious. These draft criteria were reviewed by an advisory panel consisting of a nationally recognized expert in the treatment of schizophrenia and a prominent psychiatrist from each of the organizations being studied. Criteria were only selected if the panel (1) agreed with them, (2) believed they were important in determining outcomes, and (3) thought they could be implemented with no more than slight difficulty.

Panel members considered 7 potential medication management criteria and agreed on 3 that were used in this study: (1) patients with significant psychotic symptoms should have had a change in antipsychotic medication dosage or a switch to a different antipsychotic medication within 3 months, or should have been offered treatment with clozapine (a medication with greater efficacy); (2) patients with significant akathisia or parkinsonism should have had a reduction in the dose of antipsychotic medication, a switch to a different antipsychotic medication, or a change in anti–side effect medication within 1 month, or should have been offered treatment with clozapine or risperidone (medications with fewer side effects); and (3) patients with significant tardive dyskinesia (TD) should have had a reduction in the dose of antipsychotic medication within 3 months, or should have been offered treatment with clozapine (a medication that causes virtually no TD).

There is limited consensus regarding which components of family management and case management are responsible for their effect on outcomes.^{21 ,24} Although Assertive Community Treatment is a defined, effective case management intervention,^{26 - 27} most patients do not have access to such a program, and none was offered at these clinics. Formal family psychoeducation was also rarely provided at these clinics. The quality criteria for psychosocial care did not attempt to determine whether patients were receiving effective components of family management or case management, but rather whether they received these interventions at all. The criteria examined (1) whether severely disabled patients received case management and (2) whether, for patients with close family contact, there had been any recent involvement of a family member in treatment.

A structured interview was administered to patients shortly after visiting their clinic psychiatrist. The interview included an abbreviated version of the Structured Clinical Interview for the DSM-IV,Patient Edition, Version 2.0 (SCID).^{28 - 29} Fourteen potential subjects were excluded because the SCID did not confirm a diagnosis of schizophrenia or schizoaffective disorder. The interview assessed severity of illness using the Global Assessment of Functioning scale ³⁰ ; symptoms using the expanded University of California Los Angeles version of the Brief Psychiatric Rating Scale³¹ ; akathisia using the Barnes scale³² ; parkinsonism using items from the Liverpool University Neuroleptic Side Effect Rating Scale³³ ; TD using the Abnormal Involuntary Movement Scale³⁴ ; satisfaction with treatment using the Client Satisfaction Questionnaire³⁵ ; and compliance using an established instrument.³⁶ Subjective quality of life was assessed using 1 global item from the Lehman Quality of Life Interview ³⁷ and administering it twice during the interview. Subjects were asked whether they had ever been offered treatment with risperidone or clozapine. The patient interview took an average of 40 minutes to complete (SD=9.6 minutes). An explicit, structured medical records abstraction instrument was used to collect data on medications prescribed during the past year, on documentation of symptoms and side effects, on compliance with treatment, on use of alcohol or other drugs, and on the use of treatment services. Implicit quality judgments were not made during the record abstraction. Clinical interviewers were trained to use the patient questionnaire and medical records abstraction instruments, and they had extensive training on the SCID and Brief Psychiatric Rating Scale at the University of California Los Angeles Clinical Research Center for Schizophrenia.³⁸

We operationalized the panel's criteria of "significant" symptoms and side effects using information from the patient interview. Cut points were set high enough to ensure that symptoms and side effects above the cut point would be clinically meaningful and likely to effect outcomes. Patients were defined as having significant psychotic symptoms if Brief Psychiatric Rating Scale ratings were "severe" or greater for hallucinations or suspiciousness, or "moderately severe" or greater for unusual thought content or conceptual disorganization. Patients were defined as having significant akathisia if they met Barnes'³² criteria for moderate, marked or severe akathisia; significant parkinsonism if they responded "a great deal" to items regarding muscle stiffness or slowing of movements; and significant TD if their Abnormal Involuntary Movement Scale ratings met criteria described by Schooler and Kane.³⁹ When we used this approach, 110 subjects were identified as experiencing significant symptoms or side effects.

Next, we classified subjects as receiving either poor- or adequate-quality medication management. To quality for poor-quality medication management, a subject had to meet criteria for either: A, poor-quality symptom management (management of psychotic symptoms [panel criterion 1]) or B, poor-quality side-effects management (management of akathisia, parkinsonism and TD [panel criteria 2 and 3]). Poor care was defined as either A or B since side effects and symptoms may both have a negative effect on patient outcome.

We assessed case management only for subjects who were severely ill (Global Assessment of Functioning scale score of ≤ 40, n=117) because this group would be expected to be particularly disabled and in need of case management support. Poor-quality case management was defined as present if there was no evidence from the medical records or interview of contact with a case manager during the past 3 months. Assessment of family management was restricted to patients in close contact with their families (n=153). Close contact was defined as living with the family or having contact with the family 2 or more times per week. Poor-quality family management was defined as present if there was no evidence from the medical records or interview that clinic staff had met with or spoken to a family member during the past year.

To examine whether secondary data can be substituted for more expensive primary, patient-derived information, we developed an alternative quality-assessment method that used only medical records data. We compared this "medical records method" with the main quality-assessment method described above (which used data from both patient interviews and medical records) to model the effect of restricting data sources to patients' medical records. The medical records method used the same quality criteria as the main method. However, rather than using patient interviews to assess symptoms and side effects, the medical records method defined significant symptoms and side effects as present if they had been documented in the medical records during the past 3 months.

Data were analyzed using t tests for continuous variables, χ² tests for proportions, and Pearson correlations. Two-tailed tests of significance were used and differences were considered statistically significant at P=.05. The number of schizophrenic patients in each psychiatrist's practice was estimated by multiplying the number of study subjects in their practice by the inverse of the rate at which patients at that clinic were sampled for the study.

Patient illness characteristics were similar at the 2 clinics. Subjects were severely ill with similar Global Assessment Functioning scale scores (mean=44, SD=12 at the VA; mean=43, SD=10 at the CMHC; t=.51, df=222, P=.61). As shown in Table 1, there were no significant differences between the clinics in the prevalence of significant symptoms or side effects. During the 3-month sampling frame 16 subjects (7%) had 1 visit with a psychiatrist, 167 (75%) had either 2 or 3 visits, and 41 (18%) had more than 3 visits. However, usage of new medications was quite different at the 2 clinics. At the VA, formulary restrictions made risperidone difficult to prescribe, while the CMHC had no restrictions on its use. Thirty-four CMHC subjects (31%) and 6 VA subjects (5%) were prescribed risperidone (χ²₁=25.2, P<.001). The VA clinic had a clozapine program restricting clozapine use to a limited number of physicians and to certain patients. The CMHC had a cumbersome clozapine protocol. Only 6 VA subjects (5%) and 1 CMHC subject (1%) were prescribed clozapine (χ²₁=3.5, P=.06).

As shown in Table 1, 39 subjects (17%) received inappropriate management of psychotic symptoms and 60 (27%) received inappropriate management of their side effects. Differences between clinics in the quality of symptom and side-effect management did not reach statistical significance. However, overall poor-medication management occurred more often at the VA clinic (44% vs 31%, χ²₁=4.4, P=.04). When compared with the 140 patients without poor medication management, the 84 patients receiving poor-quality medication management had a significantly lower quality of life on the Quality of Life Interview item (4.01 vs 4.64, t=3.1, df=222, P=.002), and a nonsignificant trend toward lower satisfaction with treatment on the Client Satisfaction Questionnaire (3.30 vs 3.41, t=1.5, df=222, P=.15).

Thirty-four of the 84 patients receiving poor-quality medication management had patient factors that could have contributed to this care. Specifically, 30% of poor care occurred in patients who were poorly compliant, 21% occurred in patients using alcohol or other drugs, and 6% occurred in patients with other patient-related factors, such as refusal of the recommended treatment. Poor care without contributing patient factors was significantly more common at the VA than the CMHC (28% vs 16%, χ²₁=4.7, P=.03).

Some psychiatrists had practices that included many more patients with schizophrenia and appeared, therefore, to "specialize" in the treatment of this disorder. As shown in Table 2, these psychiatrists tended to deliver better-quality care for schizophrenia, although this trend was not significant (r=0.60, P=.06). Even though postgraduate trainees (ie, residents) at the VA clinic treated relatively few patients, they had ongoing teaching from experts in schizophrenia and provided treatment quality similar to that of the most effective staff psychiatrists.

Although severely ill patients composed 52% of patients at both clinics, the VA provided significantly more of these patients with case management (Table 1). There was also a trend toward the VA providing more intensive case management, with 19 severely ill subjects (32%) receiving more than 1 contact per month, as opposed to 10 similar CMHC subjects (17%) (χ²₁=3.5, P=.06). Unlike case management, there was no significant difference between the clinics in the extent to which families were involved in the treatment process (Table 1). At both clinics more than half of the families caring for a schizophrenic relative had no communication with the clinic during the past year.

Quality assessment using patient interviews is substantially more expensive than using secondary data sources alone. However, as shown in Table 3, quality assessment that did not use patient-derived data produced strikingly different results. Relative to the main method of assessing the quality of medication management, assessment based only on the medical records identified fewer subjects as receiving poor-quality care (47 vs 84), and detected just 26 (31%) of the 84 subjects with poor-quality care. Also, the medical records method performed very differently at the 2 clinics. Differences in medical records documentation at the 2 clinics would have led to incorrect identification of the CMHC as delivering worse-quality medication management. Although both clinics documented significant symptoms and side effects at similar rates, the CMHC had a much higher rate of documentation of symptoms and side effects that were not severe enough to reach our cut off. Since the severity of symptoms was rarely documented, this led the medical records method to overstate the prevalence of symptoms and side effects at the CMHC. The medical records were of limited use in the assessment of case management and family management because there rarely was recent documentation of severity of illness or whether patients were in close contact with their families.

This is one of the first studies to measure quality of outpatient care for schizophrenia using direct patient assessment and criteria derived from national treatment recommendations. By comparing treatment guidelines with actual care, we found that poor-quality care was common at the 2 sites studied, and we identified key problems in care that might be useful targets for quality-improvement efforts. At these clinics, 38% of schizophrenic patients were receiving poor-quality medication management, 18% were severely disabled and receiving no case management, and 39% were cared for by their families without any recent contact between the clinic and family. Since we used relatively stringent criteria to define poor care, some poor care may have gone undetected. Although very little is known about the quality of care provided at other mental health clinics, we have no reason to believe that our study sites are unusual, and consequently, our study raises serious concerns about quality of care for this population. The level of treatment quality that we observed appears similar to or worse than levels of poor care reported for common medical and surgical conditions,⁴⁰ other disenfranchised populations,⁴¹ and private-sector patients with depression.⁴²

As with nonpsychiatric disorders,^{43 - 44} this study found that the choice of data source can have a powerful effect on the detection of poor-quality care, and that the accuracy of medical records cannot be taken for granted. When we relied on medical records rather than patient interviews, we were unable to identify many patients receiving poor-quality care. This occurred because significant symptoms and side effects reported to us by patients were often undocumented in medical records. Psychotic symptoms were only documented in 48% of the patients with significant psychotic symptoms, and side effects were only documented in 14% of the patients with significant side effects. Also, quality assessment based only on medical records would have incorrectly identified the CMHC as having worse-quality medication management. Although there are cost advantages to using only automated or medical records information for quality assessment, at these clinics approaches based on the medical records would have led to misleading conclusions about the quality of care.

Although we cannot distinguish whether the absence of symptoms and side effects in the medical records is due to poor documentation or a failure to detect, the low rate of documentation raises concerns about detection. At the CMHC, psychiatric treatment occurred in a private, confidential office for 20 to 30 minutes. At the VA, treatment usually occurred in "medication groups" that were 1 to 2 hours in length and attended by up to 25 patients (mean patient count=7.9, SD=4.4). Patients waited in chairs around the treatment room for their turn to briefly speak with the psychiatrist and case manager at the front of the room. It is possible that the briefer consultation and lower level of confidentiality associated with these groups reduced the elicitation and documentation of symptoms and side effects.

Our findings suggest potential quality-improvement approaches that are specific to each clinic's strengths and weaknesses. A substantial amount of poor care resulted from failure to make any change in medications over a long period. Clinician awareness of symptoms and side effects might be improved at both clinics with routine monitoring. However, in one third of the cases of poor care, symptoms or side effects were detected and documented but no intervention was undertaken. Provider education could be increased or patients directed toward psychiatrists who specialize in the treatment of this disorder. Particularly at the VA, it is possible that increased access to new medications would encourage physicians to intervene more aggressively. Both clinics should consider enhanced case management and family management. However, the effects of quality-improvement interventions such as these are not necessarily predictable and should be evaluated.

The strengths of our methods include the use of validated clinical instruments and interviewers with intensive training in these instruments. Although the methods developed here are only one of many possible approaches to measuring quality of care for schizophrenia, the efficacy literature suggests that poor care defined by our criteria leads to poor outcomes. However, several important limitations should be considered when interpreting our results. Our study focused on recent treatment, such as medication changes during the past 3 months. Since many of these patients had been treated for years, it is conceivable that all appropriate treatment strategies had been tried and found to be ineffective. However, with regard to medication treatment, the criteria recommend treatment with risperidone or clozapine for patients failing other strategies, and care was not judged to be poor if the patient had ever been offered these medications. Also, nonresponse bias can be an important problem with severely mentally ill populations. This study achieved an adequate response rate by using a visit-based sample of patients in treatment during a defined time frame. However, this design is biased toward patients in regular care. These patients may be more compliant, have less drug use, and be receiving higher-quality care than populations drawn from other clinics or the community.

Further research is needed to develop clinically useful quality assessment methods that are inexpensive enough to incorporate into routine practice. Detailed clinical assessments by highly trained staff will not be practical at many busy clinics. However, quality assessment based on the current medical records at these clinics missed most of the poor-quality care. Before quality measurement can become routine in "real world" settings, researchers may need to evaluate assessment tools that use more self-report data along with structured, automated medical records systems that interact with clinicians.

Information on treatment quality cannot only help consumers and purchasers choose between alternative programs, but it can also be used by policy makers, managers, and clinicians to identify aspects of treatment that may need to be changed. Findings from this study are consistent with the theory that treatment quality may be influenced by (1) organizational policies, such as formularies, (2) patient factors, such as noncompliance, and, (3) clinician factors, such as specialization. Policy makers may be interested in studying, for instance, how formulary restrictions or a patient's sex affect the quality of medication management. At a clinical level, this study suggests that programs providing care for schizophrenia should consider measuring and improving the quality of the treatment they provide. Research will be needed to test the effect of specific quality-improvement interventions on patient outcomes and cost if we wish to ensure that people with schizophrenia receive the best care possible.

Accepted for publication March 19, 1998.

This research was supported in part by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service as Robert Wood Johnson Clinical Scholars Fellows' Health Services Research and Development Project IIR 95-132 (Dr Young); by the Robert Wood Johnson Foundation Clinical Scholars Program at University of California, Los Angeles; by a National Alliance for Research on Schizophrenia and Depression Young Investigator Award (Dr Young); by the National Institute of Mental Health University of California, Los Angeles Research Center on Managed Care for Psychiatric Disorders (grant P50 MH-54623); and by the VISN 22 Mental Illness Research Education and Clinical Center of the Department of Veterans Affairs. Diagnostic and symptom assessment training was supported by grant MH-30911 from the National Institute of Mental Health University of California Los Angeles Clinical Research Center for Schizophrenia.

Preliminary results were presented at the annual meeting of the Robert Wood Johnson Clinical Scholars Program, Fort Lauderdale, Fla, November 16, 1996; and at the National Institute of Mental Health Conference on Improving the Condition of People with Mental Illness, Washington, DC, September 5, 1997.

The views expressed in this article do not necessarily represent the views of the Department of Veterans Affairs or of the US government.

The authors would like to acknowledge Nihua Duan, PhD, Robert Liberman, MD, Stephen Marder, MD, Elizabeth McGlynn, PhD, and Kenneth Wells, MD, for their valuable consultation to the project; and Lindsey Bergman, PhD, Leslie Tepper, PhD, and Roberto Zarate, PhD, for their contributions to collection of the data; and clinicians and managers at the clinics for their support.

Reprints: Alexander S. Young, MD, MSHS, UCLA Neuropsychiatric Institute, 760 Westwood Plaza, Los Angeles, CA 90024-1759.