The Pursuit of High Quality Accounting Standards

Abstract

My nominee for the “most incorrect belief” of accounting experts is: if accounting standard-setters have accounting expertise, constituent feedback, and political independence, they will reliably produce high-quality standards. I define “high-quality” standards as standards with social benefits that exceed their social costs.1 I argue that accounting standard-setters cannot reliably produce net beneficial standards because they lack sufficient knowledge about the causal relationships between standard-setter policy tools and social and economic outcomes of interest: hereafter, the “knowledge problem.” Judgment-based cost/benefit analysis, the way standard-setters currently deal with the knowledge problem, is a fundamentally limited solution because it relies on expertise where it is least valuable: situations involving extreme complexity and in which feedback about prior judgments is limited and unreliable.Research could potentially solve the knowledge problem, but accounting standard setting is a task for which relevant observational research opportunities are scarce and usually involve compromises that limit their informativeness. Financial reporting regulators could experiment with their policy tools to learn more about their social and economic consequences. But policy experiments such as randomized field experiments or letting firms choose from competing sets of standards would likely involve significant costs with uncertain benefits.2 The unfortunate paradox is, therefore, that we do not know enough to evaluate proposed accounting standards, and we also do not know enough to evaluate potential solutions to the knowledge problem. I conclude that, barring dramatic breakthroughs by accounting researchers or significant new policy initiatives by financial reporting regulators, mandatory accounting standards are unlikely to be of “high quality,” and how much they could be improved will remain a mystery.The remainder of this commentary is organized as follows. I begin by explaining why the knowledge problem implies that existing accounting regulators are unlikely to produce high-quality standards. I then describe some potential solutions to the knowledge problem and their drawbacks that could have limited their use so far.Accounting standard-setters recognize both the potential value of systematic, data-driven cost/benefit evaluations of proposed accounting standards (FASB 2012, 4; IASB 2010, QC35-39) and that no one currently knows enough to do them well (FASB 1991, iii; IASB 2010, QC38-39). Specifically, standard-setters know neither which types of costs and benefits matter most to diverse financial statement preparers and users, nor their magnitudes. These depend on the preferences, circumstances, and abilities of each unique financial statement preparer and user, and regulators cannot easily collect and aggregate such diffuse information (Hayek 1945; Sunder 1988). Prominent accounting and legal scholars argue that we cannot knowledgeably answer basic questions about financial reporting regulation and standardization. For example, it is unclear why financial reporting is regulated (Healy and Palepu 2001, 406; Leuz and Wysocki 2008, 2; Kothari et al. 2010, 269), whether existing regulations solve their targeted problems (Easterbrook and Fischel 1991, chap. 11; Healy and Palepu 2001, 412) without unintentionally causing new problems (Merton 1936; Kling 2010), or how the costs of capital market regulation compare against the benefits at the macro-economic level (Easterbrook and Fischel 1991, 297; Leuz and Wysocki 2008, 2; Beyer et al. 2010, 321). Research addressing such questions is “virtually non-existent” (Healy and Palepu 2001, 412).Because the knowledge problem keeps them from conducting thorough cost/benefit analyses, the FASB and IASB evaluate their own proposed accounting standards using “conscientious judgmental assessment of costs and benefits” (FASB 1991, iii). A good cost/benefit analyst would identify all of the proposed standard's material costs and benefits, forecast their magnitudes (keeping in mind how each might interact with other important outcomes), and then compare the total costs and benefits.3 How likely are accounting regulators to distinguish high-quality from low-quality accounting standards using judgment-based cost/benefit analyses? Because of the knowledge problem, we cannot directly test this question. But we can learn from the large literature exploring the powers and limits of expertise in many other domains.Experts routinely solve very difficult problems in their domain of expertise.4 But experts also frequently perform poorly, even on routine and familiar tasks (Meehl 1954; Tetlock 2005; Kahneman 2011). Kahneman and Klein (2009) conclude that a few problem characteristics explain why expert performance is either extraordinary or poor. Specifically, expertise is most valuable when (1) the task environment is “high validity,” meaning that available information signals have stable relationships with the outcomes of available actions, and (2) frequent and reliable feedback is available about the actual outcomes of previous actions (Kahneman and Klein 2009, 524). People become experts by learning from repeated challenging trials with high-quality feedback, called “deliberate practice” (Ericsson et al. 2006; Shenk 2010). Unfortunately, evaluation of accounting standards seems to be a low-validity, low-feedback task.Tetlock (2005, chap. 2) divides skeptics of expert ability to forecast complex social phenomenon into two groups. Ontological skeptics argue that inherent complexity makes high-quality forecasts impossible, whereas psychological skeptics argue that, while predicting complex social outcomes may be possible in theory, mental limits keep forecasters from using available signals effectively.Consistent with Tetlock's (2005) ontological skeptics, complex economic forecasting is needed for a thorough cost/benefit analysis of a proposed standard. A good analyst would have to forecast a long and poorly specified list of consequences, most of which cannot be reliably measured. Direct consequences include implementation costs and the value of any new disclosures to investors. Indirect and long-run consequences include strategic reactions by managers seeking to subvert new rules (Sivakumar and Waymire 2003), and increasing standardization driving good students away from accounting (Sunder 2009, 2010). Given the complexity of modern economies (Arthur 1999; Harford 2011) and the dispersion of evaluation-relevant knowledge (Hayek 1945; Kling 2010), even the best experts would be unlikely to produce the quality forecasts a thorough cost/benefit analysis requires. In addition, feedback about prior standard-setting decisions, when available, is usually unreliable. One reason is economic complexity, which confounds the effects of new accounting standards with those of concurrent economic changes. Another reason is that the delay between a policy intervention and its outcomes makes it difficult to identify the causal relations between them (Lott and Fremling 1989; Waymire and Basu 2007, 133). A process feature that limits feedback reliability is that changes in financial reporting policy frequently follow economic crises. When outcome variables differ after an accounting policy change, it is rarely clear whether the outcomes changed because of the new policy or were direct reactions to the conditions causing the policy change (Ball 1980).Tetlock's (2005) psychological skeptics identify biases that likely reduce the reliability of current cost/benefit analyses of new accounting standards. Confirmation bias is the tendency of people to favor information that is consistent with their prior beliefs (Plous 1993) and has been shown to affect accounting professionals (Cloyd and Spilker 1999; Kadous et al. 2008). Proposed accounting standards are written to persuade (Young 2003) and their costs and benefits are evaluated by the same experts who write them. Since self-evaluations are often biased (Mabe and West 1982), regulators' participation as both standard advocate and evaluator very likely (unconsciously) biases their subjective cost/benefit analyses. They thus likely overestimate the net benefits of proposed standards.5The judgment-based cost/benefit analyses of the FASB and IASB are likely insufficient to identify high-quality accounting standards because knowledge about their macroeconomic and social consequences is poor. Available methods for learning this critical information are all problematic. In the next section, I discuss two approaches that could begin to solve the knowledge problem and the drawbacks that may have prevented their use.Solving the knowledge problem requires us to learn how regulators' policy tools affect economic and social outcomes. Accounting researchers have not discovered much that helps regulators thoroughly evaluate proposed standards.6 Randomized field experiments conducted by regulators and letting firms choose from competing sets of reporting standards are two potential methods for reliving the knowledge problem.Identifying how financial reporting rules impact economic and social outcomes is difficult because it is hard to specify meaningful counterfactual worlds to compare against the real world (e.g., Ball 2008; Hail et al. 2010).7 An ideal test would compare outcomes for the same subjects both after treatment and without treatment. Because only one state actually occurs, only one can be factual (the whole group cannot both be treated and untreated). The observed group that represents the path taken is the factual group. The group representing the path not taken is the counterfactual group, and it is inevitably unobservable.Experimental and quasi-experimental design is all about specifying meaningful proxies for the unobservable counterfactual world as a way to estimate treatment effects. It is easiest to specify effective counterfactual proxies when the experiment involves: many subjects, cheap treatments, “double-blind” trials where subjects and experimenters both do not know which subjects are in the treatment group, and the opportunity to specifically assign subjects to the treatment or control groups (typically using random or matched assignment). Randomized controlled trials are considered the “gold standard” in medical research because they meet all of these criteria (Harris et al. 2001).8 Conducting research on how financial reporting standards affect social and economic outcomes is difficult because high-quality counterfactual proxies are rare. Financial reporting standards are usually applied to many firms at the same time (firms within a given industry or size range, or at the national or even supra-national level) so that a good control group is usually not available.9 Financial reporting rules cannot be changed inexpensively and are changed publicly so subjects cannot be blinded to their treatment condition. Accounting researchers do not assign subjects to experimental groups, which could compromise any experiment's internal validity.Accounting researchers have tried to specify counterfactual proxies by comparing outcomes in a given jurisdiction as accounting regulations change over time (e.g., Benston 1973; Ely and Waymire 1999), comparing outcomes across countries with differing accounting regulations (e.g., Ball et al. 2000; Atwood et al. 2011), and comparing accounting regulation to non-accounting regulations (e.g., Jamal et al. 2003, 2005; Madsen 2011). Given the research design challenges, it is hardly surprising that we have little definitive evidence about the social and economic effects of financial reporting regulatory policy (Easterbrook and Fischel 1991; Leuz and Wysocki 2008; Beyer et al. 2010).Randomized field experiments can help regulators identify causal relations between their policy tools and outcomes of interest without abstracting from or simplifying their environment. Centralized policy-makers in many contexts have used field experiments to improve their evaluation-relevant knowledge (Aigner and Hirschberg 1985; Bandiera et al. 2011; Greenberg and Shroder 2004; Karlan and Appel 2011; Manzi 2012, chap. 11; Sherman et al. 1992).The strengths of randomized field experiments are balanced by the difficulty of implementing them. They are frequently expensive and can be controversial, making them impractical in many situations.10 A randomized field experiment could be used to assess, for example, the impacts of FASB's proposals to improve hedge accounting (FASB 2011). A random subsample of firms using hedge accounting would be assigned to use the proposed standard while the other firms continue using SFAS 133. The FASB could passively observe both groups to see if managers hedge differently or firms' information environments change differentially.11While such an experiment would inform regulators about some effects of the proposed standard, it could expose firms to competitive disadvantages if the new standard differs in quality from SFAS 133. Such an experiment would likely be unpopular among the firms involved as random assignment is often perceived to be unfair (Erez 1985; Cook and Payne 2002). Consistent with random assignment being unpopular when firms are the experimental subjects, Greenberg and Shroder (2004) review several hundred field experiments, only five of which use firms as subjects, and each of which does not use random assignment but allows managers to choose whether their firm is in the treatment or control groups (Greenberg and Shroder 2004, 165–177).Letting managers choose whether to use a proposed policy reintroduces selection bias but likely reduces controversy related to fairness. While self-selection sacrifices the internal validity available through randomization, it still informs policy makers about perceptions of new policies by identifying which firms believe the new policies are desirable. Letting firms choose among several competing sets of standards (Dye and Sunder 2001; Sunder 2002, 2009; Kothari et al. 2010) would reveal managerial preferences.12 In addition, regulatory competition within a single jurisdiction relieves the knowledge problem by reducing the extent to which experts evaluate policy. When firms can choose among competing accounting standards, some evaluation is outsourced to firms that can experiment to find out which set works better (e.g., compare Microsoft and Apple operating systems). Such market feedback to competing multiple standard-setters within a given jurisdiction is a potentially fruitful solution to the knowledge problem. Markets are a low cost and effective means of aggregating dispersed knowledge about the preferences, circumstances, and abilities of market participants (Hayek 1945): in this case, financial statement preparers in firms as well as contracting parties. While likely less controversial than randomized field experiments, intra-jurisdictional standard-setter competition would still likely be controversial.13In this section, I have discussed regulatory field experimentation and intra-jurisdictional regulatory competition as potential solutions to the knowledge problem.14 Accounting research has had limited success at relieving the knowledge problem because of the difficulty of specifying appropriate counterfactual proxies. Randomized field experiments could begin to solve it but would be costly and controversial. Standard-setter competition could partly relieve the knowledge problem by providing information about firms' preferences between systems of accounting standards and outsourcing standard evaluation; though some have argued its costs could include increased risk of a “race to the bottom” and reduced comparability (Dye and Sunder 2001).15 A paradox emerges from this discussion. The knowledge problem blocks the development of high-quality accounting standards. But the potential solutions to the knowledge problem which I have explored here would likely be expensive and it is not clear whether the solutions would themselves pass a cost/benefit test.The FASB's mission is to “improve standards of financial accounting and reporting,” which the FASB claims “is accomplished through a comprehensive and independent process” (FASB 2012). I argue that the belief that existing accounting regulatory and standard-setting systems produce high-quality standards is likely incorrect because no one knows enough to evaluate whether any new or proposed standard is, in fact, an improvement. The expert-based evaluations of the FASB or IASB are unlikely to sort high-quality from low-quality proposals because the evaluation process is too complex and involves economic and social relationships that are too poorly understood. Accounting research has not, thus far, done much to relieve the standard-setters' knowledge problem. Policy initiatives, like randomized field experimentation or standard-setter competition, could begin to solve the knowledge problem, and have been effective in non-accounting fields (Sherman et al. 2002; Greenberg and Shroder 2004; Jamal and Sunder 2007; Humphreys and Weinstein 2009; Levitt and List 2009; Bandiera et al. 2011; Karlan and Appel 2011). But because little is known about the costs and benefits of these proposals, it is also not clear whether these proposals are of “high quality,” meaning net beneficial, for accounting.The way forward is, therefore, unclear. Can we conduct research on accounting standards that is evaluation-relevant? I am not sure. But if it happens, I believe that it will require us to be promiscuous, entertaining research partnerships, settings, and approaches that are not currently popular.16 In some fields, regulators have teamed with researchers to implement well-designed field experiments to try to learn which policies work and why. Similar partnerships between financial reporting regulators and accounting academics would likely be fruitful if regulators were willing to deploy their policy tools to produce evaluation-relevant knowledge.The video of this presentation, as well as the presentations for the other commentaries in this issue is available by clicking on the link below.Devils_Advocate: http://dx.doi.org/10.2308/acch-10364.s1

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 17 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.