office (03) 9820 3006 or 920 3008
fax (03) 9820 3007

Results Based Accountability (RBA)

This page reflects research carried out by Eisenbruch, Blignault, and Harris for New South Wales Health, 2004-2005, at the Centre for Culture and Health, University of New South Wales

Making the link to health outcomes via performance improvement

Over the past twenty years, there has been increasing recognition of the role that cultural competence has to play in health systems and healthcare quality (Betancourt et al., 2003; (Jones, Bond, & Cason 1998). While there are many definitions of cultural competence throughout the literature, the most cited is a variation of that proposed by Cross and colleagues (1989; in (Goode 1995) as a “…set of congruent behaviours, attitudes, and policies that come together in a system, agency, or among professionals and enable that system, agency, or those professionals to work effectively in cross-cultural situations.” Central to the definition is that cultural competence should occur across different levels – system, organisation, program, and individual - if it is to be fully effective at the level at which clients and providers interact (Brach & Fraser 2000). This multilevel approach requires a comprehensive and coordinated plan that includes interventions on levels of (1) policy making; (2) infra-structure building; (3) program administration and evaluation; (4) the delivery of services and enabling supports; and (5) the individual (Goode 1995).

  1. A major achievement for cultural competence in the U.S. has been the publishing by the U.S. Department of Health and Human Service, Office of Minority Health of the ‘National Standards for Culturally and Linguistically Appropriate Services’ (Office of Minority Health, Public Health Service, & U.S.Department of Health and Human Services 2002;U.S.Department of Health and Human Services 2001) , or CLAS,  as recommended national standards for adoption or adaptation by stakeholder organisations and agencies. Based on an analytical review of key laws, regulations, contracts, and standards currently in use by federal and state agencies and other national organizations, these proposed standards were developed with input from a national advisory committee of policymakers, providers, and researchers. In the [full report], each standard is accompanied by commentary that addresses its relationship to existing laws and standards, and offers recommendations for implementation and oversight to providers, policymakers, and advocates – see list of CLAS standard in this document.

These fourteen standards, several of which are policy mandates in the U.S., are intended to inform, guide, and facilitate required and recommended practices related to culturally and linguistically appropriate health services. These are similar in many respects to the New South Wales Ethnic Affairs Priority Statements (EAPS), in that they encourage reporting on access and equity initiatives within health service systems.

CLAS is a considerable achievement for cultural competence. However in keeping with its ontological roots the field is still evolving with ongoing efforts refining both its principles and implementation (Betancourt et al., 2002). Much of the cultural competency literature thus far has been on the importance of clinical and interpersonal aspects of medical care (Hayes-Bautista, 2003). The focus has been on cultural awareness, knowledge, attitudes and skills without necessarily describing useful steps for how a health system is supposed to become culturally competent (Brach & Fraser 2000). However, researchers, commentators and authors have recently been arguing that if the approach is to move beyond being in the eyes of many a ‘well-meaning but ultimately unproductive public relations effort’ a direct link to health outcomes must be made (Hayes-Bautista 2003).

A recent review of culturally competent healthcare systems concluded   a limitation for improving the cultural competence of healthcare systems was the absence of the link between cultural competence and effectiveness. Specifically the authors argued that this link, primarily through research, must ‘examine meaningful health outcomes and focus on what works best, where, and for whom’ (Anderson et al. 2003). The authors call for more comparative studies of culturally competent interventions to base this research evidence on.

Others suggest linking cultural competence directly to quality improvement initiatives. For example, Betancourt et al. (Betancourt et al. 2003) report experts in the field describing the need to use tools and benchmarks to evaluate outcomes that would create a standard of care for evaluation of care. This approach would directly translate cultural competence into quality indicators or outcomes that can be measured, thereby becoming a tool, in and of itself, with which to eliminate barriers and disparities. Chin (Chin 2003) argues that while service delivery systems are beginning to include statements of cultural competence in their goals or mission statements, few are operationalising these into competencies, measurable objectives, or performance indicators. Standard Nine of the CLAS standards (2000) states that ‘Integrating cultural and linguistic competence-related measures into existing quality improvement activities will also help institutionalize a focus on CLAS within the organization. Linking CLAS-related measures with routine quality and outcome efforts may help build the evidence base regarding the impact of CLAS interventions on access, patient satisfaction, quality, and clinical outcomes’ (p. 88). Other authors have also called for the inclusion of cultural competence measures in quality measure sets, arguing that the business case for cultural competence is limited by the lack of quality measurement tools (Brach & Fraser 2000). Moreover, these authors provide the link to comparative studies suggested by Anderson et al. (Anderson, Scrimshaw, Fullilove, Fielding, Normand, & Task Force on Community Preventive Services 2003), arguing that these culturally competent quality measures should be used for inter-organisational comparisons (Brach & Fraser 2000).

Further, when a system or agency works successfully towards cultural competence, cultural competence quality measures can be used for monitoring, planning and accountability purposes. An investigation into successful practices in the provision of culturally competent healthcare, conducted in the U.S. by the Department of Health and Human Services Health Resources and Services Administration (2001), nominated a number of organisations as case examples of practicing culturally competently. One of the most successful at institutionalising cultural competence was based on the inclusion of a system of accountability requiring each management team to design and implement project-based cultural performance objective plans. One of the members of the nominated agency team reported that, ‘One of the things that was the greatest step for us as an agency was the strategic planning process….We have cultural performance objective plans…The strategic plan states all managers will go through an orientation…and will select objectives for their annual performance evaluation having to do with diversity and cultural competence that they will be evaluated on, so it becomes part of their performance evaluation. That’s a way of building accountability into it, and because managers are now focussed on that, it’s made part of the team’s focused.’ (p. 25). This focus on accountability to cultural diversity has also been mentioned elsewhere as a core requirement in managed care (Lavizzo-Mourey & Mackenzie 1996) and medical ethics (Emanuel 1996).

Chin (Chin 2003) takes the accountability notion further, arguing it is the responsibility of healthcare systems to be accountable at a population health level. Such systems recognise the unique and diverse needs of both clients and community, i.e., have a population focus; ‘Performance measures driving health system quality cannot strive to attain a uniform and normative criterion without attending to differential norms and targets for different communities and racial/ethnic groups’ (p. 7).  Health disparities and inappropriate services will result among minority groups when the system has failed to be culturally competent in serving all segments of a diverse population. 

The literature linking cultural competence directly to health care quality offers a number of models that break down culturally competent health care systems into requisite parts, directly making the link to health care outcomes and reduced disparities based in health care provision. Models such as these provide a useful map of where culturally competent quality measures and indicators might sit within a healthcare system, encouraging linkages between cultural competence to effectiveness and health outcomes. For example, the review by Anderson et al. (2003) is based on a framework developed directly from the CLAS standards that flows through five culturally competent healthcare systems interventions through intermediate outcomes to health outcomes. Similarly Brach and Fraser (Brach & Fraser 2000) offer a model which links cultural competence to reducing health disparities from cultural competence techniques through clinician/patient behavioural change through appropriate services through improved outcomes to reduction of health disparities.

The state of the art is that – at least in North America – many health care organisations are developing links between performance indicators and cultural competence.

  • The Lewin group prepared for the US Department of Health and Human Services the Organizational Cultural Competence Assessment Profile (U.S.Department of Health and Human Services & The Lewin Group 2002). The Profile is an analytic or organizing framework and set of specific indicators to be used as a tool for examining, demonstrating, and documenting cultural competence in organizations involved in the direct delivery of health care and services. The outcome indicators focus on intermediate outcomes for which a plausible or credible connection/attribution to cultural competence can logically be made. To develop the Profile, the project team employed an additive process typically used in developing assessment tools that first involved the formulation of performance areas to be assessed and then the development of performance indicators for each area.
  • Siegel et al (Siegel, Haugland, & Chambers 2003) developed a set of performance indicators that would be useful in examining the achievement of cultural competence of a behavioural health care organisation (Siegel et al. 2000). Performance indicators were selected and reviewed by consumer focus groups and led to 163 indicators and 231 measures. These were reduced on the basis of importance, feasibility, and expected reliability. A Delphi procedure was used to obtain a final weighted score for each measure. The measures were reviewed against the 14 CLAS standards. The reduced set was benchmarked.
  • Various self assessment protocols for organisational cultural competence have been developed. For example, Andrulis D, Delbanco T, Avakian L, and Shaw-Taylor Y. The Cultural Competence Self Assessment Protocol for Health Care Organizations and Systems. http://erc.msh.org/provider/andrulis.pdf
  • In Europe, the WHO-HPH Task Force on MFH (Developing migrant-friendly and culturally competent health care organisations) is developing tools for assessment of cultural competence. National and regional networks of the WHO Network of Health Promoting Hospitals (HPH) played an important role in bringing the partners from Austria, Denmark, Finland, France, Germany, Greece, Ireland, Italy, the Netherlands, Spain, Sweden, and the UK together. The Amsterdam Declaration (MFH Project Group 2005), in calling for he development of migrant-friendly hospitals with quality management, provides a European-oriented policy imperative for performance indicators and cultural competence.  www.mfh-eu.net/public/files/further_mfh_activities/mfh_tf_draft_constitution_2005.pdf
  • The South Carolina Department of Mental Health Employees Performance Measurement System lists Cultural Competence as an essential objective for all employees, to include clinical staff and administrative staff at the Centers, Facilities and Central Administration.  http://www.state.sc.us/dmh/cultural_competence/cultural_plan.htm
  • The Western Interstate Commission for Higher Education APIA Cultural Competence Plan Implementation Guidelines require cultural competence performance as an integral part of the employee-provider performance evaluation system, and provider organization performance evaluation system (Western Interstate Commission for Higher Education (WICHE) Mental Health Program & Center for Mental Health Services (CMHS) 2004). Standards, guidelines and cultural competencies for managed behavioural health services for racial/ethnic populations were produced by four national panels with the recognition that in order to provide individualized mental health services, consumers should be viewed within the context of their cultural group and their experiences from being part of that group. Sixteen Principles were Cultural Competence; Consumer-Driven System of Care; Community-Based System of Care; Managed Care; Natural Support; Sovereign Nation Status; Collaboration and Empowerment; Holism; Feedback; Access; Universal Coverage; Integration; Quality; Data Driven Systems; Outcomes; and Prevention. . The Outcomes Principle requires that consumers from the four groups and their families evaluate services on the basis of actual outcomes relative to the problems that stimulated help seeking in a managed care environment. The greater the extent to which managed care plans, organizations, and providers emphasize and measure these outcomes in comparison to the expectations of consumers from the four groups, the higher the degree of consumer satisfaction.
  • The Ministry of Children and Family Development, British Columbia, Canada has developed a cultural competence framework for Contracted Service Providers and the Ministry. http://www.mcf.gov.bc.ca/publications/cultural_competency/competency_6.htm  A culturally competent system is defined against these domains: Declaration of the Importance of Cultural Competence; Policy, Practices and Procedures; Personnel Practices; Skills and Training; Service Provider Composition and Climate; Programs and Services; Community Consultation & Communication
  • The US Substance Abuse and Mental Health Services Administration lists performance measures in Cultural Competence Standards in Managed Care Mental Health Services. http://www.mentalhealth.samhsa.gov/publications/allpubs/SMA00-3457/ch3.asp
  • Cultural Competency Performance Measures for Managed Behavioral Health Care Programs. http://www.rfmh.org/csipmh/projects/id7.htm
  • Massachusetts’s acute hospital settings contracting with Medicaid in the Cultural Competence Quality Improvement Project (CC-QIP) set up monitoring of Quality Performance Indicators for Cultural Competence. http://www.diversityrx.org/CCCONF/98/ccc57.htm 
  • Dreachslin (Dreachslin 1999b) presents performance indicators for each stage of a five-part change process that she proposed in an earlier work (Dreachslin & Hunt 1996). The stages are as follows: Discovery: emerging awareness of racial and ethnic diversity as a significant strategic issue; Assessment: systematic evaluation of organizational climate and culture vis-"a-vis racial and ethnic diversity; Exploration: systematic training initiatives to improve the HCO’s ability to effectively manage diversity; Transformation: fundamental change in organizational practices, resulting in a culture and climate in which racial and ethnic diversity is valued; and Revitalization: renewal and expansion of racial and ethnic diversity initiatives to reward change agents, and to include additional identity groups among the hospital’s diversity initiatives.
  • Dreachslin (Dreachslin, Weech-Maldonado, & Dansky 2004;Dreachslin & Hunt 1996) identifies articles from the health services management literature which set forth and apply models for organizational change based on case study or survey research (Dreachslin 1999b;Motwani, Hodge, & Crampton 1995;Muller & Haase 1994;Weech-Maldonado et al. 2002) and which identify performance indicators to assess the state of organizational development vis a vis effective diversity management.

Vera (Vera 1998) notes that information for the measurement of performance indicators can be obtained from self-assessment and also administrative data (enrollment/encounter data, claims files); program and medical records; and consumer self reports. She points out that the quality of the performance measure is highly dependent on the quality of the data set from which it is derived; that the volume and complexity of data collection will also have a strong impact on the selection of performance measures; and that performance measures must be reliable and valid for their intended purpose providing similar results in comparable situations.

However, while of great importance, this work within the cultural competence industry, with the exception of Siegel et al., has focussed on developing performance indicators rather than performance measures. As mentioned by the Lewin group, this indicator development work should be seen as a ‘work in progress’ because This work is a first step along a continuum that includes further refinement of the indicators, identification of particular qualitative or quantitative measures for each indicator, identification or development of data sources and data collection instruments, and formal field testing.’ It is interesting to note that, to the best of our knowledge, this extension advocated by the Lewin Group in 2002 has yet to be published.

Cultural competence, processes and outcomes – link with Results Based Accountability

In their landmark review of where culture fits in outcomes management, Jones et al. (Jones, Bond, & Cason 1998) note that ‘all frameworks for studying outcomes possess three elements in common: structure, process, and outcome. Outcomes (the benefits of care for patients) are assessed in terms of how well the care meets or exceeds local standards of care (process), given the characteristics of the environment and its resource use (structure).  Outcome indicators are typically culture free. While differences in outcomes for different cultural groups can be studied, it is not the outcome indicators that are different but rather the variables used to select the study groups and their characteristics. Thus, differences in outcomes across or between cultural groups are typically associated with processes and structures of care’.

Several states in the United States have introduced standards and assessment tools for cultural competence (Siegel, Haugland, & Chambers 2003;Tirado 1996). In 2005 New Jersey's Acting Governor Richard J. Codey signed legislation requiring that physicians take cultural-competency training in order to obtain a license or be relicensed by the State Board of Medical Examiners. Developments such as these undoubtedly will spur research to identify the outcome of the training.

However, the question posed by the work of Jones et al. remains hanging within the cultural competence industry. How does cultural competence move from being a process oriented approach to one that links structures, processes and outcomes? The answer may lie in accountability frameworks such as Results Based Accountability (RBA). RBA is the development and use of outcomes to determine funding. RBA focuses on program results and all that can be learned from them. RBA stresses answering bottom-line questions about program effectiveness. The link between the work of Jones et al, and RBA is that RBA explicitly measures the variables that make up the process of working towards outcomes. RBA makes the process (based on local standards) of working towards outcomes (which may or may not be culture free) relevant. However, to date the critique made by Jones et al. is relevant to proponents of RBA, because to the best of our knowledge work to date using RBA has not focussed on the cultural variables that underpin culturally competent processes. However, if work does focus on cultural variables underpinning, by the reasoning of Jones et al, this work stands to uncover the ‘culture’ in outcome indicators.

There are a two examples in which cultural competence has been linked to Results Based Accountability.First, according to Careers in Social Work Education[1], cultural competence should be set out as an organizing principle, not just be one of a desired group of competencies

  • Family Support America has worked with a coalition of parents, state agencies, and community-based organizations in Colorado, Connecticut, Georgia, Michigan, Minnesota, New York, Washington, and West Virginia  to develop strategies for creating communities where families have resources to raise healthy children. Activities improved cultural competence, and developed results-based accountability in alignment with family support practices. http://www.familysupportamerica.org/content/gains/files/Seven_Year_Gains_Introduction.pdf

Despite all this activity, we found only one citation explicitly linking the Friedman framework with cultural competence – and that is by Friedman, on Promoting the Wellbeing and Monitoring Outcomes for Vulnerable Children (Friedman 2000;Friedman 2002). Under the performance measure ‘how well did we do it’ he measures timeliness of service, accessibility, cultural competence, turnover rate and and morale of staff.

RBA - How it works

In NSW a number of models currently exist for measuring performance management in health care,though none are explicitly linked with cultural competence. However, consensus has recently by human service executives reached around promoting RBA as a as a means of improving accountability, standardising the language used, and achieving greater conceptual consistency across the human service sector in measuring performance. Within health, RBA has been flagged as a robust tool for developing performance measures and results indicators which can be used to monitor and guide the Health System’s delivery of health services to culturally and linguistically communities in NSW.

The main tenet of the RBA framework is that it starts with the ends you want and works backwards, step by step, to the means by which you are going to get there. Unlike logic models of accountability, which start with inputs leading in sequence toward outcomes, the sequence of RBA is the opposite. For whole communities, the ends (also termed results or outcomes) are conditions of wellbeing for children, families and the community as a whole. For human service agencies the ends are conditions of well-being for customers of that agency, or the benefits of the services received for those who receive them. This difference between populations and clients is the key conceptual distinction lying at the heart of RBA and providing a pragmatic approach to accountability that is directly linked to outcomes. On the one hand the framework works towards ends being population results, on the other the framework works towards ends being customer results.

Population results and indicators

A population result is about the well-being of whole populations, regardless of whether anyone receives a service. They are about population quality of life, not specific programs, agencies or services such as health. An example of a population result is as simple as ‘healthy children’. The reason why population results are not about specific programs, agencies or services is that specific programs, agencies or services cannot be held solely accountable for these results. They are of course in part accountable, but to hold the NSW Health system as the sole agency responsible and therefore accountable for ‘Healthy Children’ is of course unfair and unproductive. Rather a broad number of partners and stakeholders need to become involved for achieving these population ends. The responsibility of the Health Department, where health outcomes are concerned, is therefore to assemble a team of public and private partners and creating a community strategy to make all children healthy.

Indicators measure, or quantify, population outcomes / results. Indicators answer the question, ‘How would we recognise a result in measurable terms if we fell over it?’ For example the rate of low-birthweight babies helps quantify whether the population outcome of healthy births is being achieved or not.

Client results and performance measures

Client or customer results relate to well-being of customers of programs, agencies or services. This well-being of customers is a condition that is due to that customer using the programs, agencies or services. In this respect programs, agencies or services can be held directly responsible and accountable for what they are doing, or their performance, that directly leads to the well-being of their clients. Thus the term used is ‘performance accountability’.

Performance measures measure, or quantify, performance accountability, and are the means by which the ends of customer results are achieved[2]. The most important performance measures tell whether the clients or customers of the service are better off as a result of using the service.

The link in the distinction

While these two ‘results’, and who is accountable for their achievement them, are distinct, RBA asserts it is a mistake to assume they are not related. Rather, the relationship between population and performance accountability is not linear: ‘In most previous work in this field, there is a continuous progression from population well-being to agency and performance accountability …Many programs and agencies contribute to population results by improving client results.’ (Freidman, 2000; p. 3). It is this non-liner link between performance measures and population indicators and outcomes that provide the possibility of improving quality of programs, agencies or services based on the thus far elusive link to ‘Population outcomes’.

Figure 1 below illustrates the relationship between performance measures, indicators and results across different levels of the health system. As the service system client population approaches the total population (shown by the arrow), then performance measures begin to be directly linked to indicators of population results. For the current purposes, ‘Multicultural Service’ is used to refer to those services directly offered to individual service users (e.g. Health Care Interpreter Service), ‘Multicultural Program’ refers to the all Area multicultural health services and designated staff across the State and  the ‘Health System’ refers to NSW Health as a whole. Friedman’s original framework also included a ‘facility’ level (e.g. hospital or health centre) which for the purpose of this paper will be considered as part of the ‘Health System’.

Figure 1. Relationship between indicators and performance measures (adapted from Friedman)

Using the Framework to identify Performance Measures:

Results based accountability (RBA) provides a conceptual framework to help programs, agencies and services identify key performance measures. All performance measure fit into one of four categories, derived from the intersection of quantity and quality vs. effort and effect. This intersection forms four quadrants, in each of which is ether a number # or percent data statement: 1. What did we do (effort - how much, #), 2. How well did we do it (effort - how well, %), 3. Is anyone better off? (effect, how much change - #), and 4. Is anyone better off (effect, what quality of change - %). This is shown below.

 

 

 

 

 

 

 

 

 

RBA Framework: (From Friedman, 2001)

There are a number of key points that underpin the framework. First is that clarity is required about which service, program, agency, system is being measured. This is called ‘drawing a fence’ around what is to be measured (2000; p. 7), i.e. a program, a service, a sub-program, and agency. This fence drawing exercise helps contain the focus of thinking about what’s inside the fence. Fences can also be drawn around a set of related programs or agencies that make up a service system (see also Figure One), and performance measures can be developed for the system as a whole.

Second is clarity about what is inside the fence. Who are the customers? Customers may include direct beneficiaries of the service but also include others who depend on the programs performance, like related programs and partners (for example carers or family members of patients). In NSW Health the term patient/client is used to convey this (see table one)

Third is understanding not all performance measures are of equal importance. The two general classes of performance measures that are the most important are: ‘Is anyone better off?’, telling us whether clients’ lives are better (Quadrants 3 and 4); and ‘How well did we do it?’, telling us whether the service and its related functions are done well (Quadrant 2).

‘Is anyone better off?’ (Quadrants 3 and 4) are the most important measures of all as these tell us whether clients’ lives are better as a result of receiving the service. These measures are the client or customer results. Usually RBA argues that there are four dimensions worth considering in whether clients are better off, or was there improvement or change for the better: Skills; attitude; behaviour; and circumstance.

‘How well did we do it?’, the next most important measures, include such things as timeliness of service, accessibility, and turnover rate. Friedman notes cultural competence as an example of a performance measure of performance quality in this quadrant. Friedman also notes that some performance measures leave out measures such as the cultural competence of services, ignoring this opportunity to demonstrate its value to service quality, efficacy and efficiency. However, cultural competence covers more than this quadrant, and is an input, output, process, and outcome measures (Jones, Bond, & Cason 1998).  RBA can be strengthened by explicitly recognising this breadth of cultural competence performance, and working towards the identification of culturally competent measures throughout the framework.

While not as important in developing measures, the first quadrant, ‘What did we do?’ is probably the easiest to arrive at measures with. These quantify activities performed directly relating to client results and are best conceptualised on a logic model as the outputs that services produce.

The fourth key point underlying the framework concerns the quality of the measures themselves. These require questioning on a number of levels: Communication power; proxy power; and data power. Communication power means whether the measures communicate to a broad range of audiences. Proxy power asks whether the measures say something of central importance about the program, service, or agency. Data power asks whether quality data on a timely basis can be collected (whether it is reliable and consistent).

Related to their quality, the fifth key point is to arrive at a critical number of measures, as opposed to unlimited suggestions. Friedman suggests first arriving at three to five measures that present or explain performance to policy makers or to the public. Following this are secondary measures, which are used to help manage the service, program or agency, with these often figuring in the development of baselines.

As mentioned earlier, Friedman notes cultural competence as an example of a performance measure of performance quality (‘how well did we perform the service’).   Friedman also notes that some performance measures leave out upper right quadrant performance measures such as cultural competence of services and that the use of ‘crosswalks’ as an analytic tool can be used to make these other frameworks more complete.[3] 

Subpopulations by characteristic are also commonly understood. For example, the well-being of people from a particular ethnic or cultural group might be considered as a subpopulation. And the entire Friedman process can be applied to these populations as well. For example we could put the population “All Hispanic” at the top of the page, articulate the results we want for these, how we would experience these results, identify indicators, create baselines, develop the story behind the baselines and ask who are the partners and what works to turn the curve to do better. Friedman notes, however, that the principle difficulty with results accountability for subpopulations is data. As the group gets smaller and smaller it is harder to get reliable timely data to use as indicators. This means that the process can follow the pathway where experience and the data development agenda serve as proxies for measures.[4] 

Using RBA to develop proxy measures for CLAS and cultural competence.

CLAS to date has been used to develop system indicators but not performance measures, exemplified by the work of the Lewin group (REF). However, the possibility provided by RBA or equivalent outcome focussed quality improvement work is that measure development can occur, successfully linking directly to each of the 14 CLAS standards. In this case, similarly to the potential of EAPS discussed above, CLAS serves as the uniting arrow providing cultural process variables linked to outcomes while also linking service, program and system measure development. Thus RBA provides the elusive performance measurement link between cultural competence, as outlined in the CLAS standards, and outcomes that many commentators have been calling for.