The Use of Self-Assessment in Improving the Quality of Family Planning Clinic Operations: The Experience with Cope in Africa
Pamela Lynam, Leslie McNeil Rabinovitz, and Mofoluke Shobowale
This follow-up study evaluates the effect of a clinic self-assessment technique called COPE (client-oriented, provider-efficient) on improving the quality of family planning clinic operations in Africa. In 1991 AVSC International revisited 11 clinics where it had introduced COPE 5-15 months earlier. AVSC assessed what changes had occurred as a result of the COPE intervention by (1) determining how many of the clinic problems identified by staff at the COPE introduction had been solved, (2) comparing the results of a second client flow analysis (CFA) with the initial CFA, and (3) interviewing service providers to get their opinions of the effects of COPE. Of the 109 problems identified by staff at the 11 clinics, nearly three-fourths of those that did not require external inputs were solved. In five clinics where waiting time had been a problem, average waiting time had declined 42% by the follow-up visit. The most common problems that staff were unable to solve were staff shortages and lack of private space for counseling clients. Follow-up visits and staff interviews revealed many improvements in the quality of care provided to family planning clients and increased staff involvement in solving clinic problems. COPE has proved to be a simple and effective technique that is ready to be more broadly disseminated to family planning service providers to help them improve the quality of services to clients.
AVSC International has developed a simple self-assessment methodology to help family planning service providers improve the quality of clinic operations by making them more oriented toward clients' needs and more efficiently organized for the number of clients seeking services. This technique is called COPE: client-oriented, provider-efficient.
AVSC has revisited a number of the clinics where it introduced COPE to see what changes have occurred as a result of the intervention. The results of this follow-up, presented in this paper, illustrate the variation of the impact COPE has had on clinic operations in several African countries. In many sites the time clients spend waiting for services has decreased. Several facilities have set up new channels for getting family planning supplies on a regular basis. In some cases staff have adjusted the availability of services to meet client needs--for instance, by scheduling sterilization surgery to serve postpartum clients before discharge. One hospital has set up a small dispensary in the surgery unit to make postoperative medicines available to sterilization patients. Three of the sites now have committees that meet regularly to continue the process of COPE--identifying problems and developing creative solutions.
COPE: A SELF-ASSESSMENT TOOL FOR CLINICS
COPE is both a tool for evaluation of quality of care in family planning service delivery and a tool to help improve that quality. It developed out of two AVSC pilot studies, beginning in April 1989, at 10 clinics providing voluntary sterilization in Nigeria and Kenya (Dwyer et al., 1991). AVSC has since introduced COPE at 25 sites in nine countries in Africa and at two sites in Jamaica. Several Kenyan organizations and some international organizations have also begun to introduce COPE to clinics that are associated with them.
AVSC developed COPE after concluding that the way family planning services were being delivered in Sub-Saharan Africa was itself a major obstacle to their use. Family planning clients' needs were not being met because systems were inefficient and were organized around curative services. Clients for elective services, like family planning, are not in the same position as patients who are sick and therefore willing to put up with hardship in a clinic in order to be treated. If conditions are not welcoming, clients for elective services may choose to postpone or forgo services.
Among other problems, long waiting times to see providers are common in the African clinics where COPE was started. As noted in a 1975 review of family planning programs, long waiting times are one of the most important factors explaining high rates of program and method discontinuation (Keller et al., 1975). External evaluators using traditional approaches to identify and address such problems in family planning services have not always worked very well. AVSC determined that clinic staff themselves needed to get involved in solving the problems that were hindering services.
COPE is distinctive in that it is a self-assessment tool. Rather than having outsiders--who do not know the staff, the services, or the clients--evaluate services and develop lists of problems and recommendations for improvement, COPE invites the people most knowledgeable about the clinic--the staff--to evaluate their services, identify problems, and try to develop workable solutions. The goal is to have staff ownership of both the problems and the solutions.
A nurse at one clinic in Nairobi, Kenya, commented after a COPE introduction: "If someone had come and told me about these problems, it would have been like speaking Latin to me. But since we found them out for ourselves, it's easy to see what can be done about them."
Unlike other quality-of-care assessment methodologies, such as situation analysis (Miller et al., 1991), COPE cannot be used by outsiders to evaluate a clinic. It is not meant to be used to rate one clinic against another. Rather, it gives providers a tool they can use to continuously assess and improve the quality of their own services. Precisely for this reason, the type and number of problems identified and solutions attempted vary greatly from site to site. COPE relies on staff's judgment, expertise, and experience about clinic operations; and it leaves it up to them to devise solutions to problems they identify in the analysis (Dwyer et al., 1991).
COPE is simple and adaptable to any worksite. It addresses management issues from the bottom up, and it specifically looks at services from the perspective of client convenience. Because some of the instruments were developed to look particularly at issues related to sterilization in Africa, they focus on the way services are organized to serve sterilization clients more than on clients seeking temporary contraception. But COPE does provide information relevant to temporary method use, and providers can easily adapt it to include a more detailed analysis of provision of temporary contraception. In fact, at least two health clinics (Mkomani and Bomu clinics in Mombasa, Kenya) have already adapted COPE for use in curative clinical service delivery unrelated to family planning. The methodology is also adaptable to other regions of the world.
Among family planning evaluation tools, COPE is unique in that it does not focus on outcome (e.g., couple years of protection) or distribution statistics (e.g., prevalence of methods); rather, it gives quantitative and qualitative data on the process of service delivery. One example of the emphasis on process is COPE's manual system of client flow analysis (CFA). Like the original computerized patient flow analysis (PFA) methodology developed by the Centers for Disease Control, CFA is a technique for tracking client time spent in the clinic, time spent with providers, and time spent waiting for services. In the computerized PFA, the results are presented to senior management by visiting consultants, who then discuss their recommendations with clinic management for implementation (Kopp et al., 1989). By contrast, COPE's CFA involves the entire staff from the beginning, having them log the time, graph and analyze the results, identify bottlenecks, and discuss solutions to client flow problems (Lynam, Smith, and Dwyer, 1991). Staff learn how the results are arrived at, and how to improve client waiting times.
COPE meets the key criteria for an effective evaluation methodology set forth in McNamara's 1990 review of family planning program evaluation in low prevalence countries: "multiple methods--usually both qualitative and quantitative--flexibility in research design, and the greatest possible simplicity are essential" (McNamara et al., 1990). Because COPE has several instruments, clinics can use the ones that they find most useful. If client flow is a problem, they can conduct different kinds of analyses (e.g., use of staff time, waiting time for return clients vs. new clients) to develop solutions. Some service providers have found the client interviews to be the most enlightening aspect of COPE. Several clinics, following a suggestion in the COPE checklist, have formed committees that provide ongoing forums for assessing their operations, and these facilities have found the committees to be the most useful part of COPE.
THE COPE METHODOLOGY [Footnote 1]
To date AVSC has introduced COPE at sites (clinics or hospitals) where voluntary sterilization services are well established. A COPE facilitator (trained to teach COPE to clinic staff) and staff members at the service site select three consecutive days on which to conduct the COPE activities.
The COPE technique consists of the following three main components:
On the first day, the outside facilitator tours the facility and then introduces COPE in a meeting with key clinic staff and representatives of the administration. Staff spend part of the first and second days completing the self-assessment checklist, interviewing clients, and writing up the problems identified for presentation on the third day. They spend part of the second day conducting CFA and graphing the results. On the third day, all staff again attend a meeting, during which they present to each other their findings on the program's strengths, problems, bottlenecks, and inefficiencies. After discussing these findings, they write up a follow-up plan of action. Most of the process of COPE, apart from the two group meetings, is integrated into the ongoing work of the facility, so staff spend a minimal amount of time away from their usual work during the three days.
The COPE facilitator encourages staff to conduct their own follow-up COPE exercise three months after the first one to review progress and identify new issues. The facilitator ought to return to the site after six months if possible to review progress and to make sure that staff are capable of conducting COPE on their own. The follow-up visits should be directed by a clinic staff member with assistance as needed by the COPE facilitator.
AVSC has not conducted routine follow-up visits to all sites, and the follow-up methodology has varied somewhat. AVSC is incorporating standard guidelines for follow-up visits into the COPE methodology. The six-month follow-up visit consists of the following activities:
METHODOLOGY FOR THE EVALUATION OF COPE
COPE facilitators revisited 11 sites in four African countries 5-15 months after COPE's introduction, to learn what had happened as a result of the exercise. These sites (in Ghana, Kenya, Nigeria, and Uganda) included government provincial and district hospitals, mission hospitals, university teaching hospitals, and private family planning clinics. The questions to be investigated were these:
The results presented here are the COPE facilitators' findings from follow-up visits, together with responses from staff interviews about COPE. Staff also listed responses from the client interviews they conducted as a regular part of COPE among their list of problems.
Follow-up visits served both as a way to look at the impact of COPE and as refresher training for clinic staff in doing self-assessment. Four of the 11 clinics--two private clinics and a Ministry of Health facility in Kenya, and a teaching hospital in Ghana--had done a second COPE exercise on their own before AVSC staff visited again.
FINDINGS OF THE COPE FOLLOW-UP VISITS
Three indicators describe the effects of the COPE exercise at the 11 sites: results of interviews with providers; results of a second CFA; and the number of problems that the facility had solved, or addressed, since the original COPE. An attempt to examine a fourth indicator--changes in client satisfaction as a result of COPE--was not practical at this stage of evaluation because baseline information was insufficient. However, a more formal evaluation that is under way at several COPE sites will include a comparison of client interviews before and after COPE.
INTERVIEWS WITH SERVICE PROVIDERS
During follow-up visits to some of the family planning facilities, 35 service providers who participated in the initial COPE exercise were interviewed 11-22 months later. An AVSC staff person interviewed providers at five sites in Kenya, three sites in Uganda, and one site in Nigeria, using a questionnaire with both closed (yes/no) and open-ended questions. The interviewer asked providers what their initial expectations of COPE had been, and what changes COPE had actually made in the way the clinic provides services. The interviewer also asked whether there were problems that COPE had been unable to solve, and what suggestions the providers had for improving COPE.
Provider expectations before initial COPE exercise
Most respondents recalled having had very positive expectations of COPE. Nine, however, said they had had some reservations, including fears that the exercise would be "difficult," "taxing," "tedious," "confusing," or "demanding," or that it would find fault with their work or interfere with their responsibilities. Some had thought the exercise would be of little use in the face of severe shortages of funds, equipment, and personnel.
At the COPE follow-up, all except one said their fears had been unjustified, and they had been pleased with the exercise.
Positive results of COPE
The positive outcomes of COPE most often reported in interviews with service providers included the following:
Other improvements mentioned by service providers include the posting of signs directing clients to the family planning clinic and listing clinic hours; the establishment of a numbering system so clients are seen on a first-come, first-served basis; the establishment of daily family planning lectures for waiting clients; the solution to asepsis problems; the scheduling of sterilization service daily; and a commitment to seeing high-risk women on the day they arrive instead of booking them for an appointment to see a doctor on a following day.
Problems COPE was unable to solve
Most of the problems that service providers said COPE was unable to solve require funding that has not been available. The following are examples:
Construction needs varied in scale. One site lacks the means to build a recovery area near the operating theater so patients do not have to walk a long distance to the ward; one provider said building was essential to have private space for counseling; others said they were unable to install an iron gate instead of a locked door to allow light and ventilation into the family planning area.
One provider mentioned an unsolved problem that does not require additional funding: punctuality on the part of the doctors.
Only two of the 35 service providers said they thought COPE had brought no changes to their clinic. They said COPE did not help them address the most basic problem from which all others emanate: a chronic lack of funds. Without adequate funds they are unable to purchase medical supplies, hire additional personnel, conduct formal training workshops, or carry out renovations. All staff in Uganda mentioned the funding problem and how it keeps staff morale low and makes other problems more difficult to solve.
Staff suggestions for improving COPE
Service providers made several suggestions for improving the COPE exercise, including the following:
AVSC will consider these suggestions, together with suggestions from staff and clients interviewed at other sites, as part of a more detailed evaluation of COPE that is under way. One of the outcomes of the evaluation will be recommendations for improving COPE. Almost all staff interviewed found that COPE helped them realize the importance of providing clients with thoughtful and efficient services. Some said they had recognized certain problems before COPE, but COPE gave them the forum to discuss solutions. Improved communication among staff, a better working environment, and greater motivation were all important changes that staff said COPE brought about.
RESULTS OF CLIENT FLOW ANALYSIS [Footnote 2]
Six of the 11 sites identified long waiting times for clients as a problem during the original COPE exercise. Although long waiting times are not the rule at all family planning sites in these countries, they are frequent; and where they exist they tend to be extremely long. At the sites where waiting times were a problem, waits varied from 49 minutes to 2 hours and 35 minutes.
In five sites where staff identified waiting times as a problem at the first COPE exercise, follow-up included a second CFA. [Footnote 3] Comparison of the results of the two CFAs suggests considerable improvement: waiting time declined by an average of 42%, with a range of 17 to 56%. [See Figure 1]
The solutions that staff had devised to reduce waiting times at these clinics included the following: sterilizing instruments and preparing the clinic at the end of the day instead of delaying clients while these tasks are done in the morning, doing paperwork in the less busy afternoon instead of after each client, and not requiring clients there for follow-up visits and all staff to attend the group education sessions held every morning. Some of the clinics assigned additional staff or reorganized staff duties, and one hospital director supplied more instruments because of the CFA so that more clients could be seen at the same time.
CFA gives immediate, graphic results that are often impressive to staff. Service providers are generally aware that their clients have to wait, but a frequent reaction to the results of the CFA was "We knew they were waiting, but we had no idea it was for so long." Staff members are usually very motivated to shorten waiting times when they see for themselves graphic representations of waiting times.
CLINIC PROBLEMS IDENTIFIED BY STAFF AND STATUS AT THE FOLLOW-UP VISITS
At follow-up visits to COPE sites, AVSC staff noted whether problems identified in the initial exercise had been solved. [Table 1] presents a list of the problems identified, indicates whether they were solved or not, and provides brief explanations of solutions or lack thereof. AVSC staff classified as "impossible" problems that were beyond the power of staff to solve without outside help and funds; all other problems were "solvable." The classification "attempted" denotes that staff had partially solved the problems, or at least had made serious attempts to address them. The table does not show results for individual sites, so as not to compromise COPE's status as a self-assessment tool. COPE is not intended to be comparative or punitive.
The number of solvable problems that were solved varied greatly by site; individual site proportions ranged from as low as one-third of the problems to well over three-fourths. The most important reasons for this variation are relative level of dedication of service providers; and interest, cooperation, and involvement of the administration.
Among the 109 problems identified at all sites, 58.7% were "solved." A total of 73% of the "solvable" problems were solved. Including those problems that were partly solved or at least attempted, a total of 87.5% of all "solvable" problems were solved or partly solved.
As [Table 1] shows, the problems identified by staff at a given site can range from critical issues like personnel shortages and inadequate supplies of contraceptive methods to more minor points like no tea for clients in the waiting room. AVSC abandoned an attempt to classify identified problems by level of importance because of the great divergence of opinion even among 10 AVSC staff members who tried to categorize them. This exercise was a reminder of the subjective nature of COPE: the technique is meant to motivate staff to identify and solve problems that are important to them (within the framework of meeting clients' needs) and hopefully within their power to solve. Much of the success of COPE is due to the empowerment of staff and the enthusiasm it generates by giving staff a role in improving services. Some problems may seem minor to an outsider; however, if staff can solve a few "easy" problems that have been an irritation for a long time, they may feel encouraged to tackle harder problems that have seemed out of their control, such as finding new supply routes for contraceptive supplies, as two sites were able to do.
Nearly three-fourths of the problems that were within the power of the staff to solve had been solved by the follow-up visit. Staff in many instances found simple and creative solutions. In one clinic with no privacy for counseling, staff were able to get a key to an office when it was empty and use it for counseling couples considering sterilization. (Initial counseling for women about sterilization and other methods still takes place in a corner of the waiting room.) Another site solved the problem of family planning clients' having to wait in long lines at the general hospital fee collector's office by assigning an accounts clerk to collect fees in the family planning unit.
The most frequently identified problems for which providers could not find solutions were lack of written preoperative and postoperative instructions for sterilization clients; lack of privacy for counseling; and staff shortages. Because staff at several sites identified written instructions as important, AVSC produced these, but they did not become available until after the follow-up visits.
Most of the problems judged to be beyond staff's capacity to solve had to do with funding needed for construction or transportation, or with personnel shortages. Sometimes the obstacle was an uncooperative or insufficiently interested higher-level administration. Where sites were unable to solve the problem of privacy for individual counseling, staff said construction would be required to create new space. The staff shortage was not resolved in three of seven sites where it was a problem. Staff shortages and low salaries are chronic problems in the region, and it may be to the credit of COPE that four out of the seven sites did find some solution to personnel shortages, either by internally training more staff to provide certain services (like counseling) or by convincing the administration to assign more workers to the family planning unit.
It should be noted that not all of the "solved" problems were solved entirely as a result of COPE. In some cases a solution was already in the pipeline, but COPE brought the problem into prominence by having it discussed openly and gave the extra push to get things done.
LESSONS LEARNED ABOUT COPE
Some of the important lessons that AVSC has learned from conducting COPE exercises at 25 sites and from follow-up visits are listed below.
LIMITATIONS OF THE COPE METHODOLOGY
AVSC designed COPE as an easy-to-use self-evaluation tool for family planning clinics in Africa to evaluate their services and improve the quality of care they give to their clients. The methodology is a flexible one, which can and should be adapted to the particular circumstances and needs of a clinic. Consequently, measuring the global impact of COPE, or comparing the impact from one site to another, is difficult. There is no absolute scale against which a site is measured. Instead, the progress in each site must be viewed relative to the staff's assessment at the first COPE exercise. In most of the sites followed up, staff did solve the majority of the problems they identified. And this was without needing to rely on outside donors to come through with solutions.
Because COPE is meant to be a tool for clinic staff to use to assess and improve the quality of the services they provide, organizations that use COPE should respect the confidentiality of individual clinic findings. Using findings comparatively or punitively may inhibit participants from freely expressing the problems in the clinic or from explaining progress made or not made on identified problems.
AVSC is currently conducting a more formal assessment of COPE, using more baseline and follow-up data than were available for this evaluation. We are collecting more detailed data at four sites in Africa 6 and 12 months after the COPE introduction.
Throughout its development, COPE has been very "hands-on." To make a broader impact on improving family planning operations, AVSC has two major goals for the future of COPE: (1) institutionalization--i.e., clinics should be both competent in and enthusiastic about conducting COPE entirely on their own and (2) dissemination--i.e., mechanisms should be set up to propagate COPE, perhaps through national or regional COPE coordinators in government or private clinic networks.
Institutionalization: To achieve the goal of institutionalization, AVSC has revised the agenda of the follow-up visit to assure that at least one clinic staff member has adequate skills and knowledge to direct another COPE exercise. Clinics are also being encouraged to incorporate repeat COPE exercises into their regular calendars.
Dissemination of COPE: Dissemination has begun in Africa, and AVSC will expand it to other regions now that follow-up at some of the first sites has demonstrated COPE's positive impact. All AVSC field staff in Africa have received training in COPE and are incorporating it into their program visits. COPE facilitators have also trained representatives of countrywide organizations in Kenya and Uganda, who have been introducing COPE at the clinic level.
AVSC is also completing a COPE package, which will include the COPE instruments and a manual on how to introduce COPE. The package will be targeted to staff who are to introduce it at the individual clinic level. AVSC will also begin to run workshops to train facilitators in other countries and regions.
Many organizations in the international family planning community have shown an interest in using COPE; among those using COPE techniques in some of the clinics they work in are Marie Stopes International in Africa and Latin America, Family Planning Management Development (a Management Sciences for Health project) in Kenya, and The Population Council in Kenya. The Family Planning Association of Kenya and the Christian Health Association of Kenya have programs under way to introduce COPE to all of their sites.
COPE, in summary, is an effective, simple, low-cost intervention that addresses many of the problems facing family planning service provision in the African situation. It has been very well received by service provision staff, who have used it to creatively solve--by themselves, with a minimum of outside inputs--a good proportion of the problems that confront them in their daily work.
Working Paper #2 Authors:
Pamela Lynam, M.D., is AVSC's senior advisor for medical and client-centered quality of care at AVSC's Africa regional office in Kenya; Leslie McNeil Rabinovitz, M.P.A., is a former research associate in AVSC's New York office; Mofoluke Shobowale, R.N., M.P.A., is AVSC's senior program officer in Nigeria.
The purpose of AVSC Working Papers is to capture on paper AVSC's experience and to disseminate the results of AVSC-supported research. We welcome your comments and suggestions.
The authors wish to thank all the family planning service providers who participated in the COPE exercises at their clinics. Special thanks to Antonia Scores for her invaluable assistance with staff interviews. We are grateful also to Joseph Dwyer, Pamela Harper, Terrence Jezowski, Evelyn Landry, Sylvia Sukop, and Cynthia Steele Verme for their comments on early drafts. Dore Hollander edited the manuscript. Jon Andrews designed the publication.
This publication may be reproduced without permission, provided the material is distributed free of charge and the publisher and authors are acknowledged. This publication was made possible, in part, through support provided by the Office of Population, U.S. Agency for International Development (AID) under the terms of cooperative agreement HRN-A-00-98-00042-00. The opinions expressed herein are those of the authors and do not necessarily reflect the views of AID.
440 Ninth Avenue
New York, NY 10001