No open-access location is known for this record yet.
No open-access location is known for this record yet.
The analysis and measurement of the efficiency of health care organisations is a complex undertaking. Challenges arise because such organisations pursue multiple objectives, they face operational constraints that they cannot freely choose or control, and the processes involved in producing healthcare are complex and individually tailored. This makes it difficult to specify the technologically feasible model of production and, therefore, to identify the extent to which organisations are falling short of this feasible production modality. Meeting these challenges is important: expenditure on health care accounts for a sizeable proportion of gross domestic product and citizens need to be assured that this represents good value for money. Applied academic research into efficiency has grown into a thriving industry, and we examine the supply side of the market in the first half of this editorial. The demand-side of the market, though, is less well-developed, and the reasons for this are explored in the second part. We conclude with some thoughts about how the supply of and demand for studies of organisational efficiency might be brought to an equilibrium. Judging by the increasing number of academic papers being published on the subject, the supply-side of the market in efficiency analysis is flourishing. The increase in papers is exponential: over 80% of publications are in the last decade, with the rate rising annually 1,2. Typically efficiency analyses treat a 'decision-making unit (DMU)' – for example a hospital, or a nursing home, or even a country – as the locus of production, converting inputs into valued outputs. These analyses rely on comparison of DMUs engaged in similar production processes in order to draw conclusions about their relative efficiency. A major reason for the growth in academic output is that the barriers to entering this research field have been lowered, because both the raw materials and technological processes are increasingly accessible to researchers. The primary raw materials are data for each DMU, and the last few years have witnessed substantial growth in the availability of various types of information about health care organisations and the things that they do. Most especially, in addition to the usual aggregated information about organisational costs and activities, researchers enjoy increasingly better access to detailed patient-level data. This facilitates analysis of sub-units (aka 'production lines') within multi-product healthcare organisations, such as specialties within hospitals or general (family) practices within primary care organisations. There are two main technological processes available to analyse these data, Data Envelopment Analysis (DEA) and Stochastic Frontier Analysis (SFA) 3. Increasingly, stand-alone software is available or generic computer packages are incorporating pre-canned procedures allowing researchers to apply these techniques relatively easily to their data 4. The fundamental challenge in analysing efficiency is that the true production possibility frontier is generally unknown, certainly to external observers, but probably also to DMUs themselves. This information deficit is particularly acute in the production of health care, most of which is complex and individually-tailored, and where urgent provision is often required. As Harris puts it: 'business as usual in hospital is … a continuous sequence of potential crises … {with} hospital care involving a complicated sequence of adaptive responses in the face of uncertainty … In contrast to the standardised assembly line production process, each patient receives customised attention' 5. This means that the production process is difficult to define and there is the potential for considerable variation in how and what outputs are produced. Contributions to the care process are often made by multiple agents or organisations, a 'package' of care may be delivered over multiple time periods and in different settings, and the responsibilities for delivery may vary from place to place and over time. Crucially, the researcher must ensure that the full production process is captured by the sample of DMUs under scrutiny: sound inferences about relative efficiency cannot be made unless the analysis is comparing like-for-like. But even with a like-for-like sample of DMUs, the researcher cannot observe the 'technically feasible' production frontier. Instead, an 'empirical' frontier must be identified, based on observed behaviour. DEA and SFA take different approaches to establishing the location and shape of this frontier, and to determining where each DMU is located in relation to the frontier. As such, the two technological processes generate different 'products'. There is no consensus among those who work in the field as to whether these products should be considered complements or substitutes. Many researchers specialise in one technique, thereby avoiding the issue. Those that employ both techniques have to provide an explanation when – as is frequently the case – the two techniques yield different DMU-specific estimates of inefficiency. SFA takes an indirect approach to measuring inefficiency. Stochastic frontier models control for supposed influences on output, and contend that unexplained variations in output are due – in part, at least – to inefficiency. Unlike standard econometric models, therefore, where interest lies in the explanatory variables, stochastic frontier models extract DMU-specific estimates of inefficiency from the 'unexplained' part of the model. The implication is that standard econometric tools to test model specification cannot be applied to SFA models both because of the interpretation placed on the residual and because DMU-specific rather than average estimates are required. Thus un-testable judgments need to be made about the adequacy of stochastic frontier models and the inefficiency estimates they yield 6. Advocates of DEA would argue that the problems of model specification can be avoided by allowing the data to 'speak for themselves'. In applying DEA the location and shape of the frontier are established empirically. The outermost observations, those with the highest level of output given their scale of operation, are deemed efficient. Consequently, DEA is highly flexible, the frontier moulding itself to the data. The drawback, though, is that the frontier is sensitive to DMUs that have unusual types, levels or combinations of inputs or outputs. These will have a scarcity of adjacent reference observations, perhaps resulting in sections of the 'frontier' being inappropriately positioned 7. While DEA might be thought to win out over the SFA method in terms of flexibility, this is offset by how the technique interprets any distance from the frontier. There are two key differences. Firstly, DEA assumes correct model specification and that all data are observed without error while SFA allows for the possibility of modelling and measurement error. Consequently, even if the two techniques yield an identical frontier, SFA efficiency estimates are likely to be higher than those produced by DEA. Secondly, DEA uses a selective amount of data to estimate each DMU's efficiency score. DEA generates an efficiency score for each DMU by comparing it only to peers that produce a comparable mix of outputs. This has two implications. First, if any output is unique to a DMU, it will have no peers with which to make a comparison, irrespective of the fact that it may produce other common outputs. An absence of peers results in the automatic assignation of full efficiency for the DMU under consideration. Second, when assigning an efficiency score to a DMU not lying on the frontier, only its peers are considered, with information pertaining to the remainder of the sample discarded. In contrast, SFA appeals to the full sample information when estimating relative efficiency. In addition to making greater use of the available data, this facet of the SFA approach will make the sample's efficiency estimates more robust in the presence of outlier observations and to the presence of atypical input/output combinations. But this advantage over DEA is mainly a matter of degree - the location of (sections of ) the DEA frontier may be determined by outliers, but outliers also exert influence on where the SFA frontier is positioned. Moreover, there are no statistical criteria for sorting these unusual observations into 'outliers' or 'examples of best practice' 6. Competition to publish has encouraged innovation and product differentiation among researchers, who are exploiting the longitudinal nature of many datasets 8,9, applying increasingly sophisticated analytical techniques in order to extract as much information from the data as possible, for example decompositions of allocative and profit efficiency 10, providing confidence statements around efficiency estimates 11–13 and attempting to provide insight about the drivers of inefficiency 14. All this has led to an increase in both the volume and quality of output among the academic community. Although there has been an increase in the supply of these studies, it is not apparent whether they are satisfying an expressed demand. There are three distinct consumer markets for this type of analysis. Academics, of course, need to keep abreast of latest technological advances in the field, and many papers and conferences on the subject are targeted exclusively at this audience. However, two other markets appear ripe for exploitation – policy makers and the DMUs themselves. Attempts have been made to produce efficiency analyses intended to impact on policy, the most high profile being the WHO 'World rankings' 15. Although the WHO exercise was highly controversial, some policy makers are using efficiency studies to inform their decision-making. Two concrete examples are the use by Queensland Health in Australia, where DEA is used on a sample of hospitals, among a suite of performance indicators (see http://www.health.qld.gov.au/quality/mq_reports.asp for more information), and in Norway where efficiency analyses were used in 1995 as one argument for moving to DRG-based financing 16 and are currently used by some regional Health Enterprises to inform their internal resource allocation (see, in Norwegian only: http://www.tidsskriftet.no/lts-pdf/pdf2005/3300-2.pdf). Yet these policy applications are few and far between. If efficiency studies are not part of the routine analytical arsenal of policy makers, why not? Policy makers require analytical tools to satisfy key conditions, including relevance, timeliness and reliability. Efficiency analyses should address relevant concerns of policy makers. In most health systems, the pursuit of efficiency remains a central preoccupation, arising from concern about the health care expenditure and the need to structure incentives such that DMUs are encouraged to pursue efficient behaviour. In competitive markets these incentives are taken as given: short-term inefficiency erodes profit margins, long-term inefficiency predicts exit. But health care markets are rarely competitive so, in the absence of policy intervention, inefficiency may be persistent. DEA and SFA appear to offer policy makers insight into the variation in efficiency among DMUs and, thereby, of where efficiency gains might be made. Although efficiency analyses are inevitably retrospective, this is not a unique handicap – most performance monitoring relies on historical data. The ease with which DEA and SFA can be applied implies that only the speed of the information flow impedes the timeliness of analysis. The main reason that efficiency analyses are little utilised by policy makers appears to stem from concern about their reliability. As explained above, because DEA and SFA construct the unobservable efficiency frontier and calculate distances from this construct in different ways, the estimates of relative efficiency produced by each technique usually differ. Estimates are also often sensitive to choices about how to formulate SFA and DEA models, particularly so when analysing complex production modalities such as in health care. This sensitivity undermines confidence in whether the analytical tools can estimate DMU-specific inefficiency reliably. Policy based on unreliable evidence is best avoided. Until this fundamental issue is resolved, marketing efficiency analyses to policy makers will yield limited returns. What might efficiency analysis offer DMUs to encourage their 'buy in' to the efficiency analysis market? There are three considerations that limit active participation by DMUs: incentives to pursue efficient behaviour, incentives to participate in comparative exercises, and specificity of the information. The incentives for DMUs to seek to improve the efficiency of their productive processes may have become sharper. As public health systems move from historical budgeting to some form of case-related payment mechanisms and as patients are given greater choice about where to receive treatment, the health care market has become more competitive. These competitive pressures should spur organisations to pursue greater efficiency and expand market share. The take-up of efficiency analyses might increase in consequence. Efficiency analyses are comparative, but depend on accurate information. Where there is limited disclosure of information, any form of benchmarking exercise will be partial, and offer limited insight. In competitive settings, a DMU would like to know about their competitors without yielding much information about themselves. More informative analysis may be feasible in regulated industries where organisations might be mandated to provide the requisite information. But, of course, mandatory disclosure does not preclude inaccurate reporting 17. Participation in such exercises might be secured if they are conducted by independent research organisations and the results are anonymised 18. Even were there to be accurate disclosure, efficiency analyses may not provide DMUs with the requisite information to take action. Efficiency analyses tend to focus on the organisation as the unit of analysis, but this may provide DMUs with little insight about where technical improvements should be made. This is particularly problematic for DMUs, such as health care providers, that have multiple production lines. Instead specific information is required about the efficiency of these different production processes. In the past, the poor specificity of information and lack of concrete recommendations for action would appear to be the greatest barrier to DMU buy-in to the use of SFA and DEA. If the health care market were perfectly competitive, measurement of (in) efficiency would be unnecessary. But inefficiency probably persists in imperfect markets, and DEA and SFA have been developed to identify the extent of inefficient behaviour. The supply of efficiency analyses is impressive. However, the demand-side of the market for efficiency analysis is under-developed. Most analyses appear to be targeted at academic consumers, and are little utilised by either policy-makers or DMUs themselves. For greater penetration into the demand-side of the market, the analytical techniques need further development. Two aspects of product development are particularly important. First, there needs to be greater confidence that the analytical results are reliable. This necessitates greater analytical attention to model construction, especially clearer definition and measurement of the nature and quality of healthcare outputs and better understanding of the production process and of production constraints. Second, the analysis needs to be more specific. Instead of merely quantifying the extent of inefficiency, analyses need to identify the nature and form of inefficiency. Neither of these challenges is straightforward, and academic researchers need to work more closely with policy makers and DMUs in order to meet them. Bruce Hollingsworth holds a Victorian Health Promotion Foundation Research Fellowship and Andrew Street receives funding from the English Department of Health. We are grateful for reviewers' comments on an earlier draft. The views expressed are those of the authors only.
Loading…
Stored only when both papers are in the corpus — 88 citations exist upstream.
Papers that cite the same references — the relational fallback while embeddings are still backfilling.
Imported from OpenAlex (CC0) — nothing here is generated.