Critical Evaluation of The Theory and Practice of Evidence-based Policy

It is often noted that there is something intrinsically appealing about the concept of evidence-based policy (EBP) (Shillabeer et al., 2015). Indeed, the antithesis, opinion based policy, is often seen as objectionable (Davies, 2004). Inherent to these sensibilities, however, is a preconceived definition of ‘evidence’. For example, prevalent understandings of ‘evidence’ include positivist and empirical modes of input-output observation (Green, 2000). In contrast, normative and theoretical reasoning is often disregarded as an appropriate evidence base (Sarat and Silbey, 1988:107) although it remains an essential and pervasive source of information for policy (Zane and Welsh, 2017). ‘Evidence’ it seems, is a highly ambiguous and broad term. It will be suggested that these ambiguities are central to the disparity between the theory of EBP and its implementation and practice. Further, implicit in this disparity, is the question of whether EBP is first, possible and secondly, desirable. To begin, a review of definitions and understandings of evidence bases, and their conflicts will be provided. The implications surrounding current conceptions of the stages model of the policy process will follow. Thirdly, the practice of EBP will be discussed in a sectoral comparison between medical and penal policy. The importance of networks and a ‘guideline’ for appropriate evidence selection will be emphasized throughout.

Conflicts between theory and practice are not unique to policy studies. Indeed, such tension can be found in political (Ozcelik, 2006), mathematical (Malara and Zan, 2003) and aesthetic (Borgdorff, 2012) arenas also. However, techniques of conflict resolution in these areas remain undeveloped (Ozcelik, 2006; Wells, 2004). For example, in the context of policy, Green (2000) and McLeroy et al. (1993) suggest multiple theories may be required in tandem to successfully develop programmes and policies. However, the absence of a ‘guideline’ or ‘rationale’ for selection may significantly hinder the success of a policy (Green, 2000:126; Kriesberg, 1995:170; Buchanan, 1994). In this way, the theoretical foundation for EBP can be seen as incomplete: while there may be support for the integration of evidence into policy, there is limited guidance on how the most effective evidence should be selected and assimilated.

Furthermore, Davies (2004) notes that, firstly, the uncertainty of knowledge and, secondly, the different status of knowledge fields, present significant challenges to EBP. In other words, the conflict of status between positive and normative knowledge, and the lack of a coherent strategy of conflict resolution between these contribute to an unstable theoretical foundation for EBP (Zane and Welsh, 2017). That is, the value placed on empirical research often neglects the essential function of normative research (Zane and Welsh, 2017). One result is a ‘knowledge vacuum’ where the surplus and preference for positivist evidence fails to advance policy as alternative forms are neglected (Davies, 2004: 4). Indeed, Bullock et al’s (2001) report concluded that government departments use a limited range of evidence from a narrow set of resources.

Further, ‘uncertainty of knowledge’ may arise from the ambiguity surrounding the definition of evidence and, as suggested above, what form it should take (Nutley et al., 2002). For example, the Oxford English Dictionary (OED) defines evidence as a ‘body of facts or information indicating whether a belief or proposition is true or valid’ (OED, 2018). However, this definition is limited in two ways. Firstly, there is ambiguity surrounding the existence of an objective ‘truth’ free from social and cultural ideological bias (Nath, 2014). Evidence should, therefore, be considered ‘contingent and contextual rather than universal, determinant and invariable’ (Buchanan, 1994:274). This questions the impartiality associated with positivist evidence and the natural sciences (Buchanan, 1994; Green, 2000). Parkhurst (2017:7), for example, notes that technical bias may arise from the neglection of rigorous scientific practice in order to provide evidence in support of certain political interests. Both normative and positivist evidence can thus produce equally biased outcomes, questioning the ability of evidence to provide any certain ‘validity’.

This is closely linked to the second limitation: implicit in the OED definition is the separation of ‘facts’ and ‘information’. As Zane and Welsh (2017) suggest, positivist understandings (‘facts’) remain largely descriptive while normative evidence (‘information’) tend towards prescriptive claims - while they differ by intention and methodologies, it is imperative the two are recognized as inextricably linked. Otherwise, the potential of EBP may be limited to ‘the way the world is’ rather than ‘the way the world ought to be’ (Monahan and Walker, 1988:467). The use of normative evidence, for example, integrates non-observable facts, such as equality and fairness, into policy (Faigman, 1999). Moreover, the theory can be used in the evaluation of the policy to address a variety of inappropriate variables (Green, 2000:126). However, the exclusion of positivist evidence from the policy process in exchange for normative can result in maintenance of power and social structures that are harmful to social equity (Sarat and Silbey, 1988:107). Moreover, as will be discussed further, it is essential to policy formation in areas such as medicine (Chalmers, 2003). The two, it seems, are mutually dependent although this remains to be established in official policy rationale (Nutley et al., 2002:2). Again, the absence of a ‘guideline’ to inform appropriate evidence selection can exacerbate this conflict of status.

Similarly, there is a conflict between propositional (i.e. formal and academic knowledge) and non-propositional (i.e. informal and obtained through experience) evidence (Malone et al., 2004:83). On the one hand, the latter emphasizes the conditionality of evidence collection: normative evidence based on non-propositional knowledge can be limited by the number of individuals and their combined life experiences (Malone et al., 2004:83). Consequently, networks combining skills and experience are integral to the expansion of the evidence base (Nutley et al., 2002; Bowen and Zwi, 2005). This may help mitigate the gap between theory and practice as researchers and actors can use non-propositional knowledge to create alternative pathways (or ‘guidelines’) to navigate appropriate evidence use (Bowen and Zwi, 2005). Furthermore, the potential for non-propositional, normative ‘information’ to be converted into propositional, positivist ‘fact’ (i.e. theory generation) emphasizes both the importance of partnerships and the interdependency of the concepts (Titchen and Ersser, 2001). However, such reasoning is somewhat idealistic as evaluation practices differ in funding, capabilities, and rigour, therefore, limiting the potential for extensive evidence assimilation (Head, 2016). A broader interpretation of ‘evidence’ should thus be developed (Green, 2000); one that incorporates the entire spectrum of evidence and recognizes the improbability of finding an objective ‘truth’ (Higgs & Titchen, 1995). If an extended definition of evidence were to be adopted, it may be argued, the foundations for EBP would become stabilized thus aiding its translation into practice. Such conditionality is summarised in Nutley et al’s (2002:2) four requirements for improving EBP:

Agreement as to what counts as evidence and in what circumstances

A strategic approach to the creation and accumulation of robust evidence

Effective dissemination of evidence and wider access to knowledge Initiatives to ensure the integration of evidence into practice

Similarly, Weiss (1998) suggests that for EBP to be successfully implemented methodology and results must be undisputed and support existing political ideologies, that policies have strong ‘champions’ and the consequences reversible and robust (Nutley, Davies, and Walter, 2002; Weiss, 1998). If these are the optimal conditions for EBP, it thus follows that policies constricted by a narrow understanding of evidence are not providing an adequately broad theoretical basis for the practice of EBP (Parkhurst, 2017). Indeed, ‘Modern Policy Making’ (NAO, 2001) concludes that use of a variety of knowledge sources and analytical skills is essential for risk management and the success of EBP.

Moreover, further theoretical incoherency may arise from inadequate conceptualization of the policy process. That is, a simplistic conception of the policy process obscures the point at which evidence informs policy, complicating the translation of EBP into practice. For example, proponents of the linear model of the policy process suggest that policy formation begins with problem identification (Araral, 2012). Here, it is assumed that evidence would, and for some should (Andrews, 2017), inform policy formation from its infant stages.

Further, feedback loops and the ‘policy evaluation’ stage implies that the process is continually informed by evidence (SOAS, 2018; Sutton, 1999) implicitly suggesting that evidence provides the impetus for policy formulation (SOAS, 2018). Indeed, the model’s utility and pervasion are often attributed to its heuristic function and simplistic idealism (Roe, 1991) thus mitigating the need for accurate, in-depth description (de Leon, 1999).

However, the evidence is not utilized so exhaustively (Richards, 2017) and the model’s simplicity has provoked the development of more complex theories. For example, in recognition of the possibility of externalities affecting the implementation or revision of a policy (i.e. a ‘policy window’), Kingdon (1999) posits a more complex model. Drawing on Cohen et al’s (1972) ‘garbage can’, Kingdon recognizes that organizations, problems, solutions, and processes are non-linear and anarchic (Cairney, 2015). As a result, when problems arise, an organization may draw on a mix of solutions from the ‘garbage’ - i.e. a selection of resources yet to be assimilated into a solution that could be either ineffective or preferable (ibid). The time at which the solutions in the ‘garbage can’ may be used, however, remains undetermined and therefore reflects the reality of evidence used in the policy process: information and facts may be introduced at any stage and to an uncertain end (ibid*). Moreover, Kingdon’s (2003) description of a ‘policy primeval soup’ recognizes that policies are continually evolving due to factors other than evidence, for example, the strength of networks (Nutley et al., 2002; Bowen and Zwi, 2005).

However, while Kingdon offers significant progress in the stages model, conceptions of the policy process continue to hinder the theoretical foundations of EBP in two ways. Firstly, simplistic models of policy formulation continue to pervade and thus contribute to an inaccurate portrayal of evidence use - one where it is seen to dominate the policy process (SOAS, 2018; Sutton, 1999). Secondly, models fail to provide a causal account of how evidence enters the policy process (Howlett et al, 2014:10) and thus hinders the formulation of a ‘guideline’ for evidence use.

Nonetheless, the development of more complex theories undermines the assumption that evidence is integral to the policy process. This has implications for the use of the term ‘evidence-based policy’. Indeed, in recognition of the often subordinate use of evidence in policy, there is a preference for the term evidence-informed policy - despite the changing agenda (Head, 2016; Nutley, Davies, Walter, 2002).

The concept of EBP gained significant traction in the late 1980s and 1990s with the rise of conviction politics (Nutley, Davies and Walter, 2002). Specifically, Labour’s Modernising Agenda (1997) centralized EBP in a bid to move from ‘what works’ to ‘why it works’ (Wells, 2004:6). However, some equate the rise with technocratic policy-making and a managerialist agenda (Sanderson, 2002). Furthermore, policies generated by this movement can be seen to neglect additional elements informing policy formulation (Parsons, 2002).

For example, the economic climate and political context can determine access to resources affecting policy implementation (Head, 2016). Such factors affecting policy are conceptualized by Kingdon (1999) as three streams (policy, problem, and politics) which interact and, in the correct ‘window of opportunity’, converge to produce policy. For example, it has been suggested that EBP has developed in tandem with a political climate of accountability in which taxpayers demand the efficient use of fiscal resources (Myers and Spraitz, 2011; Weiss et al, 2005:28). To give the example of policing; between 2010 and 2014, the ‘problem stream’ of taxpayer demand and limited fiscal resources converged with the ‘political stream’ of austerity resulting in 25% reduction in funding (NAO, 2015: 4). However, evidence suggests these cuts may undermine the statutory basis for networks and increase demands on the police force, thus having adverse effects on crime control (Karn, 2013:21-31). The political socioeconomic context, therefore, is given equal, if not more, consideration than evidence in policy formulation. As Davies highlights in Labour’s agenda; policy is not just a matter of ‘what works’ but ‘at what cost’ (2004:5).

However, similar to the normative versus positive debate above, this example should not be indicative of the polarization between ideology and evidence. For example, Chang and Wang’s (2016) study of the relationship between political discourse and empirical evidence suggests they are inextricably linked: the use of empirical research in policies is designed to make causal inferences and are thus essential to the normative reasoning of political rhetoric. Further, the interplay between the two is suggested to be integral to the functioning of open, democratic societies (Davies, 2004:5; Dillow, 2014). This begs the question; to what extent should evidence inform policy if other factors are of equal importance?

Certainly, there is no correct ‘ratio’ for policy formulation and the egalitarian use of resources and evidence type is not necessarily preferable across all public sectors (Nutley, Davies and Walter, 2002; etc etc**). Medicine, for example, has established a ‘hierarchy of evidence’ which prioritizes systematic reviews and randomized experiments on the one hand, and devalues political ideology, observational studies and professional consensus on the other (Hadorn et al, 1996; Nutley, Davies, and Walter, 2002:3). Two reasons for this may be suggested. Firstly, there is a consensus as to the desired outcome of health policy (reduced mortality and greater health) thus facilitating methodological selection (Nutley et al, 2002:3). Secondly, the consequences of the less rigorous methodology are significant: reduced treatment effectiveness and thus adverse health (Moher et al, 1998; Nutley et al, 2002).

Indeed, a greater status assigned to positivist evidence in medical policy has led to theoretical coherence across health care and the establishment of beneficial practices (such as the integration of patient values into physician’s behavior) and institutions (such as the Cochrane Collaboration) (Sackett, 1996). Consequently, EBP has minimized the effects of bias and chance (Chalmers, 2003). Moreover, while this does not necessarily prevent the dissemination of methodologically weak evidence, the systematic approach to evidence selection and standardised practices for critical evaluation (for example, by the Cochrane and Campbell Collaboration) offers a marked example of optimal EBP practice (Davies, 2004): ‘what counts’ as evidence is readily understood and guidelines are offered for theoretical coherence enabling successful translation into practice.

However, its practice is not easily transferred to other sectors. For example, the introduction of the ‘What Works Centre for Crime Reduction’ in 2013 aimed to ‘improve the way government and other organizations create, share and use high-quality evidence for decision-making’ (Hunter et al., 2016: iv). However, seven limitations have been identified, three of which will be expanded. Firstly, it was noted that there is ‘no well-developed road-map’ offering a guideline of evidence used in practice (Hunter et al., 2016: xi). Significantly, this was attributed to the limited development of a theoretical foundation (Hunter et al., 2016: xi). Secondly, the report identified that limited resources prevent the optimal use of ‘evidence champions’ and the successful creation of networks (Hunter et al., 2016: xiii). Thirdly, they highlighted the need for ‘consistency in messages across [...] various evidence mechanisms’ (Hunter et al., 2016: xiii). These three limitations contrast with EBP in medicine and conflict with Nutley, Davies and Walter’s (2002) four requirements for EBP, namely:

A strategic approach to the creation and accumulation of robust evidence Effective dissemination of evidence and wider access to knowledge Initiatives to ensure the successful integration of evidence into practice

Despite centralization on the political agenda, the use of evidence, it seems, fails to be successfully translated into practice across sectors. Indeed, the success of EBP initiatives in education and social care have procured equally ambiguous outcomes (Bellamy, 2013). This questions the possibility that EBP can, and even should be translated into practice across all sectors due to reasons of theoretical inconsistency.

Finally, it is worth noting that while the theory and practice of EBP still contain significant limitations, this should not be taken as support for its abandonment. For example, the Police Research Group (an inter-agency research project) tasked with the reduction of burglary rates illustrates the potential of evidence to inform practice (Laycock, 2000). Analysis of crime rates concluded that the majority of burglaries were repeated in the same location. Consequently, a task force was established to protect already vulnerable houses and, in one example, repeat victimization fell to zero and burglary rates decreased by 75% (Laycock, 2000). The clarity of a desired outcome and methodology, appropriate funding and compatibility with political ideology at the time can be seen to have ensured this success. However, especially in the context of criminal justice, the creation of evidence is often biased in support of a specific ideological justification for punishment (Cohen, 2012). Further, the absence of a guideline for appropriate evidence selection leaves successful EBP like the example above, in a minority.

The theory of EBP can thus be seen to rest on unstable foundations, hindering its translation into successful practice. Indeed, ambiguity in both the definition of ‘evidence’ and the status of fields of knowledge have been shown to be detrimental to theoretical coherence. Moreover, the polarization of positivist and normative, propositional and non-propositional evidence is pervasive - despite their mutual dependency. Further, simplistic concepts of the policy process imply that evidence is the primary impetus for policy formulation. However, additional factors such as political ideology and economic climate have been shown to be of equal significance. The presence of political ideologies and bias question the extent to which EBP is indeed in opposition to opinion based policy and whether ‘evidence-informed policy’ would offer a suitable, and more accurate, alternative. Moreover, the sectoral comparison indicates that, while successful examples do exist, the enforcement of EBP may not be appropriate across all fields. Thus, for the theory of EBP to be successfully translated into practice a broad and coherent framework for evidence selection and integration must be established. Like policies, the ‘guideline’ should make full use of all evidence types, as well as networks and partnerships. Moreover, the flexibility and inclusivity of this theoretical approach need to be adopted in practice in order to enhance its utility across all sectors. Thus, while EBP may indeed be possible, current theoretical understandings limit its value in practice.