URE Methods: Thinking in the Field
Here you will find selected resources developed to help users explore current perspectives on URE methods. This section focuses on work about methods used to study URE, not the methods themselves.
If you would like to recommend resources for inclusion in this catalog, please send suggestions to contact@uremethods.org.
Examination of Methods
These publications provide overviews and commentary on methods used to study URE.
Social science has done well in providing empirical studies that depict how research is used in policymaking. Yet it has performed less well in another contribution science can make—developing explanatory theoretical frameworks that predict and promote future research use (Ness, 2010). For those who conduct theory-driven research studies, Carol Weiss (1999) raised a provocative question two decades ago that remains relevant today: Have researchers been groping around in the dark looking in the wrong places for the wrong purposes for the wrong reasons? Weiss claimed that some research contributions are not “easy to see” and may “not be visible to the naked eye” (Weiss, 1986, p. 217).
Existing studies have focused disproportionately on the supply side of research utilization—the conduct and communication of research to policymakers—with far less attention to the demand side—the uses to which policymakers put research (Gamoran, 2018; Tseng, 2012). To shed light on the demand side, Elizabeth Day, Emily Parrott, and I revisited existing theory by turning to policymakers themselves for their perspectives on how research contributes to policymaking. We compared how policymakers said they used research with the predictions of four prominent theories of research utilization. What we learned is that Weiss’s words may have been prophetic: If researchers rely only on existing theories to measure research use, they may be missing what policymakers see as important contributions to their decisions and to the policy process.
Citation: Bogenschneider, K. (2020). Fresh Insights on Measuring Research Use: Policymaker Perspectives on How Theory Falls Short. New York: William T Grant Foundation.
As we consider the state of domestic policymaking in the United States, it is easy to feel disillusioned about the role research evidence has played over the last few years. From denialism to misuse, science seems to be frustratingly politicized to the detriment of a healthy and equitable society. In many ways the policy and scientific communities have never felt more disconnected.
A bright spot for us during this time, with support from the Foundation, has been the opportunity to test a model that would bridge these two communities—ultimately improving the use of research evidence (URE) in child and family policymaking.
As we consider the state of domestic policymaking in the United States, it is easy to feel disillusioned about the role research evidence has played over the last few years. From denialism to misuse, science seems to be frustratingly politicized to the detriment of a healthy and equitable society. In many ways the policy and scientific communities have never felt more disconnected.
A bright spot for us during this time, with support from the Foundation, has been the opportunity to test a model that would bridge these two communities—ultimately improving the use of research evidence (URE) in child and family policymaking.
Read the full article here: http://wtgrantfoundation.org/congressional-use-of-evidence-can-be-improved-reflections-from-a-trial-of-the-research-to-policy-collaboration-model
The phenomenon of confirmation bias has received increased attention in the current era of partisan politics. Defined as “the tendency to process information by looking for or interpreting information that is consistent with one’s existing beliefs,” the term has been used to describe the resistance of individuals on one side of the political spectrum to the arguments of the other. We recently found ourselves reflecting on a similar tendency, which we observed in our work introducing new instructional practices in schools. Of course, we don’t suggest that we encountered resistance per se, but rather an inclination to fit what is new into what is familiar.
The project, Cultivating Excellence in English Learner Instruction (CEELI), was designed to bring evidence-based practices for improving English-learner literacy into classroom routines in five medium-sized urban or suburban school districts. Introduced to relevant research by scholars and experts at an initial convening, participating district teams then identified a specific instructional strategy, based on the research, that they wanted to tackle. Districts worked together as a networked improvement community (NIC) to implement their strategies, gathering virtually on a monthly basis over a six-month period. The work of the NIC was intentionally classroom-based. The goal was to see how research actually made its way into classroom practice versus how it was designed to do so.
Read the full article here: http://wtgrantfoundation.org/avoiding-confirmation-bias-when-implementing-evidence-based-instructional-practices
In Studying the Use of Research Evidence: A Review of Methods, Drew Gitomer and Kevin Crouse highlight measures and methods from a range of methodological traditions that have been employed by researchers to assess the use of research evidence in disparate policy and practice domains, including education, child welfare, and public health. The report outlines core methodological issues in the study of the use of research evidence, reviews recent studies that illustrate specific data collection and study design methodologies, and discusses the affordances and limitations of each.
For each method, Gitomer and Crouse examine what they call “threats to valid interpretation,” which researchers would do well to consider as they design new studies. The authors also identify the distinct research questions that each methodological approach is best poised to answer. Importantly, Gitomer and Crouse give particular attention to measurement: how do we know research evidence is being used when we look for it? This challenge poses a significant obstacle to progress, and we hope that this report will help researchers identify the variety of approaches available and better understand their strengths and limitations.
Citation: Gitomer, D. & Crouse, K. (2019). Studying the Use of Research Evidence: A Review of Methods. New York: William T Grant Foundation.
An increased focus on the use of research evidence (URE) in K-12 education has led to a proliferation of instruments measuring URE in K-12 education settings. However, to date, there has been no review of these measures to inform education researchers’ assessment of URE. Here, we systematically review published quantitative measurement instruments in K-12 education. Findings suggest that instruments broadly assess user characteristics, environmental characteristics, and implementation and practices. In reviewing instrument quality, we found that studies infrequently report reliability, validity, and demographics about the instruments they develop or use. Future work evaluating and developing instruments should explore environmental characteristics that affect URE, generate items that match up with URE theory, and follow standards for establishing instrument reliability and validity.
Citation: Lawlor, J., Mills, K.J., Neal, Z.P., Neal, J., Wilson, C., & McAlindon, K. (2019). Approaches to measuring use of research evidence in K-12 settings: A systematic review. Educational Research Review, 27, 218-228.
These examples suggest at least two recommendations for those studying the use of research evidence in education policymaking. (We include others in our book.) The first speaks to how to assess the role of intermediary organizations. Second, we recommend that studies of the use of research evidence include policy analyses in addition to examining research focused on the functioning and effects of educational interventions. After a long period of dormancy, the study of research evidence use has made remarkable advances over the past decade. Consequently, it is an opportune time to strengthen the validity of URE policy studies by expanding the range of evidence analyzed.
Read the full article here: http://wtgrantfoundation.org/expanding-the-definition-of-evidence-in-studies-on-the-use-of-research-evidence-in-policy
Measuring the use of research evidence (URE) by schools has become a central focus of education researchers. However, it has proven challenging due to low response rates, social desirability bias, and costly or time-consuming data collection methods. To overcome these challenges and meet the needs of research focused on URE, this paper introduces a non-reactive archival measure: Archival Search of Use of Research Evidence (ASURE). ASURE counts references to research or evidence on a school’s or school district’s website to capture the extent of its rhetorical use of research evidence. After illustrating the collection of ASURE in all public school districts in Michigan (N = 595), we use data on these districts to show that ASURE is reliable and valid, and thus offers a promising new strategy for measuring URE in schools. We conclude by considering future steps for exploring ASURE, not simply as a measure of URE in schools, but instead as a measurement strategy for assessing URE in a broad range of organisational contexts.
Citation: Neal, Z., Lawlor, J., Watling Neal, J., Mills, K., & McAlindon, K. (2019). Just Google it: measuring schools’ use of research evidence with internet search results. Evidence & policy: a journal of research, debate and practice, 15(1), 103-123. https://doi.org/10.
Because it can be hard to pin down exactly how someone’s ideas have shifted in response to research, it’s often difficult to see the conceptual uses of research.
But as part of a study of research-practice partnerships in three districts, and through our work developing measures of research use in the National Center for Research in Policy and Practice, we have identified three approaches that can help us see the uptake of research ideas in practice. Each approach relies on a different data source, including interviews, observations, and surveys.
Read the full article here: http://wtgrantfoundation.org/study-conceptual-use-research-consider-tradeoffs-among-methods
Improving the use of research evidence in ways that benefit youth requires clarity not only about what counts as quality evidence, but also what counts as quality use. Surprisingly, the question of what it means to use research evidence well remains largely unexplored, even amid wide-ranging international efforts to strengthen the role of research in social change. In response to this need, the authors have developed the Quality Use of Research Evidence (QURE) Framework. It characterizes high-quality use of research evidence as thoughtful engagement with and implementation of appropriate research evidence, supported by a blend of individual and organizational enabling components within a complex system.
Read the full article here: http://wtgrantfoundation.org/toward-quality-use-as-well-as-quality-evidence
Further progress on our study of what research looks like and how we can promote research use among policymakers and practitioners is impeded by the persistent challenge of studying and measuring use of research evidence. Elizabeth Farley-Ripple has argued that research use remains rather elusive as a target of measurement largely because it is not well defined or fully connected with actual practice. If the use of research evidence is inherently about what users do with evidence, then it makes complete sense to study it as a behavior. The most obvious advantage of doing so is that we already have valid and reliable tools—theories, models, frameworks, methods, and measures—to measure and analyze behavior in a systematic way. These can be rather easily adapted to create measures of the who, what, why, how, when, and where aspects of evidence use that are comparable across different actors, settings, and circumstances as well as over time.
Read the full article here: http://wtgrantfoundation.org/study-use-research-evidence-behavior
Critical Perspectives on URE
These publications provide Critical Theory perspectives on URE and have significant import for the methods used to study URE.
A new era of theory furnishes the explanatory frameworks it will take to make researchers more effective across policy settings. Community Dissonance Theory conceptualizes researchers and policymakers as two separate communities that confront different questions of interest, employ different decision-making processes, and speak different languages (Bogenschneider & Corbett, 2021; Bogenschneider, Corbett, & Parrott, 2019). The theory’s foundational premise is that researchers can more effectively engage policymakers if they begin with the basics of better understanding the policy community—its inhabitants, institutions, and culture.
Read the full article here: http://wtgrantfoundation.org/engaging-policymakers-a-new-era-of-research-and-theory-that-builds-on-the-basics
The William T. Grant Foundation has sought to support new research on ways to promote the use and usefulness of research evidence to improve youth outcomes for over a decade. They have funded numerous studies aimed at identifying and testing strategies to improve the use of research evidence (URE) in policy and practice decisions that affect young people. In the URE focus area, definition is key. They have defined research evidence as a type of evidence derived from applying systematic methods and analyses to address a predefined question or hypothesis. This includes descriptive studies, intervention or evaluation studies, meta-analyses, and cost-effectiveness studies conducted within or outside research organizations. As a human endeavor, research is inextricably implicated in the societal structures and systems that have served to maintain power hierarchies and accept social inequity as a given. Indeed, research has been historically and contemporaneously “(mis)used” to justify a range of social harms from enslavement, colonial conquest, and genocide, to high-stakes testing, disproportionality in child welfare services, and “broken windows” policing (Au, 2016; Jerrim & de Vries, 2017; National Academies of Sciences, Engineering, and Medicine, 2018; Wells, Merritt, & Briggs, 2009). Critical perspectives offer possibilities for repairing these wrongs and for reimagining the possibilities of what research can accomplish.
In this essay, Fabienne Doucet asks: What possibilities do such perspectives illuminate for rethinking the production of research with the objective of making research more useful and relevant? Here the idea of usefulness takes on a different quality than it has in the past because it interrogates and centers how well research evidence communicates the lived experiences of marginalized groups so that the understanding of the problem and its response is more likely to be impactful to “the community” in the ways the community itself would want. She argues that this shift requires focusing on the production of research and the ways it can be reimagined. Put another way, “improving the use of research evidence” will require ensuring usefulness–and use–toward ends that are congruent with the goals and visions that marginalized communities have for their self-determined benefits (Tuck & Yang, 2014).
Citation: Doucet, F. (2019). Centering the Margins: (Re)defining Useful Research Evidence Through Critical Perspectives. New York: William T Grant Foundation.
Among researchers and others in the field who examine the use of research evidence in policy and practice, questions of how and what have been the primary locus of inquiry. Considerably less attention is directed toward questions of why, who, and for whom research is used. From a racial justice perspective, to not raise these questions from the beginning can be thought of as an exercise in dismissal— the hardline of indifference that is not about remembrance or amnesia but the power inchoate in not having to recognize, acknowledge, or grapple with the forces of history (cf. Alexander, 2012).
It would be naïve to consider the use of research evidence a neutral act. The (mis)use of research evidence, from test scores to skull sizes to “validated” assumptions about what constitutes beauty, has been used to construct the ideology of race—to set in motion the racial hierarchy that both elevates and centers Whiteness while simultaneously reducing and criminalizing Blackness. The (mis)use of research evidence has seen Black bodies reduced as the object of White oppressive fetish (e.g., the Black body as empirically sexualized and contrived as abnormal, a monstrosity, etc.), preoccupation with social control (e.g., the Black body as target of sterilization), and experimentation (e.g., the Black body as experiment for White medicine).
That is, attempting to conceptualize the use of research evidence without critical attention to why, who, and for whom that evidence is used misses a vital truth: The use of research evidence is not only embedded in systems of power, it is a system of power.
Citation: Kirkland, D. (2019). No Small Matters: Reimagining the Use of Research Evidence From A Racial Justice Perspective. New York: William T Grant Foundation.
This paper looks beyond the environmental and sustainability education (ESE) field for ideas on understanding the research-policy relationship. It examines two specific bodies of literature that have analysed the interplay of research and policy in different ways – critical policy studies and evidence use studies. Bringing these two literatures into conversation, we draw out insights in relation to: what counts as evidence in policy decision-making, what influences policy processes beyond evidence, and what roles research can play in relation to policy making. We then consider how these issues from beyond the field might advance research and policy in ESE. We argue that ESE policy is distinctive in its scale, breadth and contestation, and that there is a need for more diverse work in relation to the ESE research-policy interface.
Citation: Rickinson, M., & McKenzie, M. (in press). Understanding the research-policy relationship in ESE: insights from the critical policy and evidence use literatures. Environmental Education Research. https://doi.org/10.1080/13504622.2020.1804531
Implications of Studying URE
These publications provide overviews of URE research with implications for the methods used to study URE.
The use of research evidence (URE) in policy and practice is relevant to many academic disciplines, as well as policy and practice domains. Although there has been increased attention to how such evidence is used, those engaged in scholarship and practice in this area face challenges in advancing the field. This paper attempts to “map the field” with the objective of provoking conversation about where we are and what we need to move forward. Utilizing survey data from scholars, practitioners, and funders connected to the study of the use of research evidence, we explore the extent to which URE work span traditional boundaries of research, practice, and policy, of different practice/policy fields, and of different disciplines. Descriptive and network analyses point to the boundary spanning and multidisciplinarity of this community, but also suggest exclusivity, as well as fragmentation among disciplines and literatures on which this work builds. We conclude with opportunities for to improve the connectedness, inclusiveness, relationship to policy and practice, and sustainability of URE scholarship.
Citation: Farley-Ripple, E.N., Oliver, K. & Boaz, A. Mapping the community: use of research evidence in policy and practice. Humanit Soc Sci Commun 7, 83 (2020). https://doi.org/10.1057/s41599-020-00571-2
The scientific rigor of education research has improved dramatically since the year 2000. Much of the credit for this improvement is deserved by Institute of Education Sciences (IES) policies that helped create a demand for rigorous research; increased human capital capacity to carry out such work; provided funding for the work itself; and collected, evaluated, and made available the results of that work through the What Works Clearinghouse. Major challenges still remain for education research, however. One challenge is dealing with the replication crisis that has plagued other scientific fields and is likely to be a problem in education science. A second challenge is better supporting the generalizability of education research. A third challenge is adapting our rigorous research designs to the increasing complexity of our interventions and our questions about the mechanisms by which these interventions achieve their effects. Promising approaches to meet each of these challenges are suggested.
Citation: Hedges, Larry V. (2017). Challenges in building usable knowledge in education. Journal of Research on Educational Effectiveness, 11(1), 1-21.
Research can play a vital role in pointing policymakers, civil society, and communities toward a stronger, more sustainable, and just world. But getting there means building on what we know about what it takes for research to be useful, used, and impactful.
Citation: Tseng, V. (2022). Research on research use: Building theory, empirical evidence, and a global field. William T. Grant Foundation.
In this chapter from What Works Now: Evidence-informed Policy and Practice, Vivian Tseng and Cynthia Coburn delineate the evolution of evidence use in the United States, specifically the ways that research evidence has been used in public education. From the narrow focus of the What Works agenda to the emergence of more holistic conceptions of research to improve education, evidence use in the U.S. has followed a winding path through changing landscapes. While top-down, one-way strategies have given way to greater mutualism in recent years, what will it take for this trend to continue? Here, Tseng and Coburn offer three recommendations that comprise a fresh framework for thinking about evidence use.
Citation: Tseng, V. & Coburn, C. (2019). Using evidence in the US. In A. Boaz, H. Davies, A. Fraser, & S. Nutley (Eds.) What works now?: Evidence informed policy and practice (pp. 351-368). Bristol, UK: Policy Press.