Second Round of Online Course Starting February 5th: Using Research to Inform Decisions
Hosted by the Harvard Graduate School of Education, NCRPP is offering an online course February 5-18, 2018 that will help district leaders, school leaders, and state education officials learn how to find, evaluate, and use research to inform the decisions they make.
Learn from the co-principal investigators of NCRPP who are leading scholars in the field:
- Heather Hill, Jerome T. Murphy Professor in Education at the Harvard Graduate School of Education
- Bill Penuel, Professor of Educational Psychology and Learning Sciences at the University of Colorado Boulder
- Derek Briggs, Professor of Research and Evaluation Methodology at the University of Colorado Boulder
- Cynthia Coburn, Professor at the School of Education and Social Policy at Northwestern University
How to Register
- Dates: February 5-18, 2018
- Cost: $149 per person
- Register: gse.harvard.edu/ppe/rid
Federal guidelines now ask school and district staff to use research evidence when making decisions about policies and programs. Despite these realities, finding and using solid research can be a difficult task.
Using Research to Inform Decisions will guide you in how to find and evaluate pertinent research, with a constant eye toward how such research can be used to inform various school- and district-level decisions. The workshop provides an overview of how practitioners use research and offers several simple tools to help you find the study you need. You will also learn strategies and structures that help foster an environment where research truly informs decision-making.
This two-week online workshop is jointly led by HGSE Professor Heather Hill, University of Colorado Professor Bill Penuel, and other leaders in the field. Video lectures, readings, facilitated online discussions, and exploration exercises will help you engage with research and gain confidence in using evidence to make sound decisions.
By completing the program, you will be able to:
- Explore the nature of research use and current national patterns in research use
- Learn how to find relevant research and read it accurately
- Evaluate research design, and be able to judge the credibility of the research
- Implement strategies for increasing the use of evidence within your own educational setting
The online instruction, discussions, and job-embedded practice should take five to seven hours over the duration of the program. Participants who complete all individual assessments and contribute to group discussions will receive a certificate indicating completion of five clock hours of instruction.
*Note: all content is self-paced, including group discussions. Facilitators will moderate discussion boards throughout the two-week program.
A Descriptive Study of the IES Researcher-Practitioner Partnerships Program
For a related webinar hosted by NCRPP with leaders from IES-funded researcher-practitioner partnerships, click here.
This report presents the results of a descriptive study of the Researcher–Practitioner Partnerships in Education Research program, a two-year grant program funded by the Institute of Education Sciences at the U.S. Department of Education that supports exploratory research within a partnership context. In these partnerships, researchers collaborate with practitioners from at least one state or local education agency on a research project to investigate a problem of practice and to identify strategies to address the key issues.
The descriptive study focused on the first three cohorts of researcher–practitioner partnerships (RPPs) using a mixed-method, cross-case design. It relied on three sources of data: surveys, interviews, and original grant applications. A total of 106 participants from 27 of the 28 funded RPPs completed the survey, the interview, or both.
The most commonly reported goals of the RPPs were ones emphasized within the request for applications (RFA) for the program: conducting and using research, and impacting local improvement efforts. RPPs also pursued the goals of cultivating partnership relationships, increasing the capacity of researchers and practitioners to engage in partnership work, and informing the work of others through the sharing of strategies and tools with people outside the partnership.
Partnerships reported the greatest progress in their initial work toward the goals of building relationships and refining their understandings of problems the partnership could address. They reported somewhat less progress for longer-term goals, such as improving organizational policies and processes in the educational organization and impacting students’ outcomes.
Conducting and Using Research
The majority of partnerships focused on descriptive or exploratory questions, and some focused on investigating causal relationships or validating measures or constructs.
Most proposed to use mixed methods and to draw on both existing and new data sources. To carry out these research plans, partnerships created multi-sector datasets, engaged in secondary analyses of student-, teacher-, and school-level data, conducted interviews and focus groups, created case study reports, and more.
Activities and Communication
All partnerships were engaged in problem refinement and analysis of data, and nearly half were doing some design work together that involved developing programs and practices that they were testing or planning to test.
Participants in the RPPs maintained regular and frequent communication across partners, with most members engaging in weekly emails or phone calls.
Roughly half of the partnerships identified organizational turnover and obtaining usable data as key challenges to their work together.
Other leading challenges were synchronizing schedules of researchers and practitioners and accommodating different timelines for getting work done.
Perceptions of Partnership
Researchers and practitioners alike highly valued their participation in partnership work, with 100% of those surveyed either agreeing or strongly agreeing that they would participate in an RPP in the future.
Practitioners named several benefits, including helping to shift organizational culture for research use and increasing access to resources and expertise aimed at understanding and addressing a specific problem of practice.
Both researchers and practitioners also felt that the quality and applicability of research increased as a result of the partnership.
Findings from a Survey of State Science Leaders
This report presents the findings of our study of research use among state science leaders, led by Megan Hopkins, assistant professor of Education Studies at the University of California, San Diego.
We surveyed members of the Council of State Science Supervisors (CSSS), a professional organization composed of science education specialists who serve at the state, territorial, or the protectorate educational agency in the United States and U.S. territories. Its members include all current supervisors of science education in State Education Agencies, as well as former supervisors. Within their jurisdictions, supervisors play key roles in directing efforts at ensuring excellence and equity in science education as outlined in A Framework for K-12 Science Education, which guides the development of science standards in many states and was used in the articulation of the Next Generation Science Standards.
The study found that networks of policy makers and educational leaders facilitate access to research that informs decisions related to statewide science education reform.
USING RESEARCH IN IMPLEMENTATION EFFORTS
We asked CSSS members a variety of questions to understand the activities they engage in as state science supervisors, as well as how they access and use research in their work. Sixty-one CSSS members across 38 states responded to our survey and reported that their work most frequently involves activities that attend to key components of their state’s education systems, including: reviewing and developing state science standards, designing state science assessments, and designing or conducting teacher professional development.
The research that CSSS members used to inform their state’s decisions focused on instructional practices and student learning in science, such as the Ready, Set, Science! report by the National Academies Press, and the edited volume, Working with Big Ideas. Very little of the research that CSSS members used, however, focused on science education reform in diverse contexts.
LEVERAGING NETWORKS FOR RESEARCH USE
CSSS members most often accessed research through their professional networks, including through CSSS, the National Science Teachers Association (NSTA), and colleagues in state departments of education, rather than via other research outlets such as the What Works Clearinghouse. Broad engagement in CSSS networks, such as through CSSS-sponsored meetings and workshops, appeared to facilitate the use of research more than activities like visiting or collaborating with other states.
These professional opportunities likely facilitated the development of a robust research network for CSSS members, who reported turning to a variety of individuals for research to inform their state’s decisions related to the Framework. These individuals represented a range of organizational affiliations, and included 16 current CSSS members, 7 former CSSS members, 8 state department of education leaders, 7 local educational leaders, 29 individuals working for educational non-profit organizations, and as many as 75 university researchers.
CSSS MEMBERS’ RESEARCH NETWORK
CSSS members had a well-connected research network, indicating that most individuals could access one another via other individuals in the network, such that research could potentially flow across the network relatively easily. A few former CSSS members and university researchers were prominent in CSSS members’ research network, suggesting that the research they shared was particularly influential. Former CSSS members also were key sources of research, suggesting that their participation in CSSS activities helped to preserve association memory around the kinds of research that inform members’ work, which is important for sustaining implementation efforts.
Survey of Practitioners’ Use of Research (SPUR)
The instrument used to survey a nationally representative sample of school and district leaders, called the Survey of Practitioners’ Use of Research (SPUR), is now available for use by those interested in research use.
The NCRPP research use survey focuses on the core of “research use” in three related, but separate, constructs: the ways that school and district leaders 1) apply research in their decision-making processes; 2) value research as an important component for decision-making; and 3) evaluate research quality. The survey instrument also includes measure of organizational culture for research use and open-ended reports of research that educational leaders find useful.
The survey can help researchers describe and compare the prevalence of different kinds of research use among district and school decision makers and be useful for evaluations of actions intended to improve research use.
An extensive survey development process to establish the validity and reliability of measures. Two different groups of advisors with practical and research expertise in the use of research reviewed items. The team conducted cognitive interviews with 40 educational leaders to ensure items were comprehensible and to determine whether items elicited the focal constructs. We assessed the internal consistency of scales used and used item response theory to evaluate scales’ ability to discriminate amongst different respondents. Overall, all of the scales showed good reliability. Information about the survey development process, reliability of scales, and findings can be found in Technical Report #1 (see below).
Findings from a National Study on Research Use Among School and District Leaders
This technical report presents results of a nationally representative survey of principals and district leaders in the nation’s mid-sized and large school districts conducted by the National Center for Research in Policy and Practice. A total of 733 school and district leaders responded to the survey from 45 states and 485 school districts.
The survey focused on how these educational leaders use research to inform their decision making, identifying three types of research use defined in earlier studies: instrumental use of research to guide or inform a specific decision; conceptual use of research to inform the way a person views a problem or possible solutions; and symbolic use of research to persuade others or legitimate a decision already made.
School and district leaders also answered questions about their attitudes toward research, their efforts to acquire it, and the culture of research use in their organizations. Finally, an open-ended survey item asked school and district leaders to name a specific piece of research they found useful.
Types of Use
- Respondents most commonly reported instrumental use of research. Common forms of instrumental use were to design professional development for teachers and administrators and direct resources to programs.
- With respect to conceptual use, 71% of respondents indicated that the research they encountered had expanded their understanding of an issue.
- Among the symbolic uses presented to respondents, leaders most frequently reported using research to convince others or using research selectively to support a decision.
Sources Leaders Used to Obtain Research
- Leaders most frequently accessed research through professional associations, professional conferences, and colleagues.
- Leaders less frequently accessed research directly through individual researchers or from the What Works Clearinghouse, the National Center for Education Statistics, or the Regional Educational Laboratories. These sources overlap with professional associations, conferences, and colleagues, such that these findings indicate points of access, rather than relative influence.
Attitudes Toward Research
- Leaders endorsed the idea that research is relevant to practice, but indicated that the time lag between conducting and publishing research can decrease its usefulness.
- Respondents reported very positive attitudes about the value of educational research, with nearly all endorsing the ideas that research can address practical problems and that researchers provide a valuable service to educational practitioners.
- Leaders were mixed in their perceptions of the credibility of research, with half saying that researchers could be biased and about a third saying that researchers often framed their results to make a particular point.
Effort to Acquire Research
- Although a majority of leaders said they would look for research to inform a new problem or decision, few said they would contact researchers directly.
Knowledge of How to Interpret Conclusions from Research
- The majority of responses indicated understandings of the role of purposeful sampling in qualitative research and how to interpret effect sizes in quantitative research.
- Few leaders drew accurate conclusions about what can be learned from a case study and half of respondents did not respond accurately to a question about random assignment.
Culture of Research Use
- Although most leaders reported that research is viewed as a useful source of information in their district or department, a majority disagreed with the statement that people expected claims made in meetings to be backed up by research.
Specific Pieces of Research Leaders Found Useful
- The pieces of research that school and district leaders named as useful were most often books, research or policy reports, or peer-reviewed journal articles.
- Most named research focused on instructional practices and learning in the classroom, though few mentioned research pertaining to specific subject matter content areas.
- Respondents most often claimed to use research named to support their own learning, inform the design of programs, and provide instructional leadership.
Findings from a Survey of Research Attitudes and Use in the Nation's Largest Districts
In spring 2015, 271 central office staff members from 32 of the largest districts in the U.S. completed a survey about their attitudes toward research and how they use research in their work.
This report shares the results of that survey overall and by four types of professional roles in the district: Area & School Supervision; Curriculum & Instruction; Special Education; and Assessment, Accountability & Research.
District leaders viewed research as valuable, credible, and relevant.
The overwhelming majority reported that education research was valuable (95%), credible (85%), and relevant to their work (87%). However, participants were split regarding whether there was a “disconnect” between the worlds of research and practice.
District leaders used research to inform decisions, expand understanding, and persuade others.
Almost all reported using research to inform decisions (99%). In addition, all participants reported using research to inform their thinking or expand their understanding of an issue (100%), and almost all reported using research to persuade or convince others of their position (96%).
The pieces of research that district leaders named as useful were most often books, research or policy reports, or journal articles focused on instructional practices.
When participants were asked to name a piece of research that had been useful in the past year, they most frequently named books (36%), followed by research or policy reports (29%) and peer-reviewed journal articles (23%). Most pieces focused on instructional practices, but did not focus on a particular content area or subgroup of students. Similar to the various types of research use reported, participants most often claimed that the pieces they named were useful for developing their own professional learning and knowledge (29%), followed by providing instructional leadership for others (25%) and designing policies and programs (20%).
High-quality educational research that could shed light on effective policies and practices is increasingly accessible to districts and schools. Since its establishment in 2002, the Institute of Education Sciences (IES) at the U.S. Department of Education has funded dozens of field- initiated efficacy and scale-up studies of interventions, released multiple evaluation studies of major policy initiatives, supported rigorous studies of programs through the Regional Educational Laboratories, and funded training grants to prepare new scholars to conduct more scientific research in education. Efforts such as the What Works Clearinghouse, Doing What Works web site, and syntheses of research presented in practice guides, are all efforts to increase access to scientific research.
Providing access is only the first step to improving student outcomes. Districts and schools must then use this research to improve the quality of learning opportunities they provide to students. At present, there is a small body of inquiry on how district and school leaders use research. The results from such studies are consistent: research is rarely used, and not in the linear fashion imagined by most (Coburn, Honig, & Stein, 2009). Research use involves interactive processes, including contention, persuasion, negotiation, and sensemaking (Amara, Ouimet, & Landry, 2004; Contandriopoulos, Lemire, Denis, & Tremblay, 2010; Earl, 1995). It requires leaders to make sense of conclusions, deliberate about their relevance to the current context, and create policies that reflect agreements about what the research suggests they should do in that context.
We need to understand much more about the interactive processes involved in research use, or what Tseng (2007) has called “the demand side” of research use before we can improve district and school leaders’ research use. We need measures of research use to track progress at both the central office and school levels. We also need to understand better how school and district leaders currently use research. It is only by understanding how these local leaders actually make decisions, and the role of research in this process, that we can begin to design interventions that promote more effective uses of research. And, finally, we need more research on existing strategies to promote research use. Today, there are a number of efforts to foster stronger partnerships between researchers and practitioners as a strategy to increase research use (Coburn, Penuel, & Geil, 2013), but little is known about the role of partnership research in decision making.