IES Research Use Studies

This study examines how district leaders use research in their instructional decision-making in four large urban districts. We focus on organizational routines, which are a central medium through which instructional decisions are made in school districts. Routines may matter for research use because they bring particular people together at particular moments in the decision-making process, shaping what and how decisions are made, and likely the role of research therein. However, organizational routines have received little attention in existing scholarship on research use.

For this report, we focused on 140 interviews with district leaders that related to district organizational routines around ELA professional development in four urban school districts.

• How, if at all, do organizational routines structure decision-making?
• What research and other forms of information do district leaders use in their decision-making?
• How does the structure of organizational routines shape the role of research and other forms of information in decision-making?

• In all four districts, district leaders accomplished the complex work of instructional decision-making around ELA professional development by using multiple, interrelated, routines that both divided up decision-making into different tasks and also connected decision-making between individual routines.
• Districts divided-up the disparate work of decision-making into three different types of routines: design, deployment, and diagnosis.
• All four districts drew on common sources for information, including data, research, individuals, and organizations. But, each district had distinct portraits of information use in their decision-making. They varied in the number of distinct information sources (range), the relative distribution of information types (balance), and the degree to which a fewer or larger number of district leaders invoked research and other forms of information (spread).
• The type of the routine, whether it be design, deployment, or diagnosis, influenced what information district leaders used.
• The presence of connections or lack of connections between routines shaped the degree to which research use in one routine influenced the work in other routines within a district.
• We uncovered a new form of research use, which we call latent use. Latent use occurred as district leaders in one routine embedded research in artifacts, which then guided the work of leaders in other routines in substantive ways.

This report describes a study of the activities and influence of three different types of research-practice partnerships (RPPs) that shared a common focus on improving mathematics teaching and learning. The three types of RPPs were a networked improvement community (NIC), a design research partnership (DRP), and a research alliance (RA).

The findings are derived from analyses of 63 observations of partnership meetings, 58 different products, 122 surveys, and 133 interviews across the three RPPs.

• How did the RPPs differ with respect to their organization and major activities?
• What kinds of research products did the RPPs produce?
• What influence did the RPPs have on partner districts’ decisions regarding policies, programs, and practices?
• What influence did the RPPs have on sharing research-based ideas with their district leader partners?

RPPs overlapped and differed in their organization and major activities. All engaged in research activities, with two also engaging in design research practices, such as developing, testing, and iteratively refining an innovation designed to improve teaching and learning outcomes, and one engaging in improvement science practices, including Plan-Do-Study-Act cycles.

The three RPPs produced multiple research products. The RPP using a NIC approach most commonly produced and shared research- based improvement strategies, while the other two RPPs most commonly shared their work and findings at academic conferences. It was common in two of the three RPPs for research and practice partners to co-author products together.

RPPs influenced districts’ decisions and research use. Education leaders active in all three of the RPPs reported that their partnerships’ work influenced the design of professional development in the district. RPPs influenced other policies and practices in the involved districts as well, including providing new frameworks for thinking about mathematics, and integrating new practices modeled in the partnership into the district.

Education leaders reported that they consulted their research partners in some key decisions. RPPs differed with respect to which decisions they were consulted on, however, across decisions related to topics including professional development design, scaling up a program, and program resource allocation.

Within these broad findings were important contextual differences related to the nature and structure of each RPP. Click the image above to download the full report and read more about findings and recommendations.

This report presents results from a study of the Council of State Science Supervisors (CSSS). CSSS is a professional association for state science leaders that works to sustain and nurture a dynamic learning community and to empower its members to be effective and articulate advocates for quality science education at the local, state, and national levels.

Given that many state science supervisors are working to support implementation of the Framework for K-12 Science Education, we sought to develop an understanding of CSSS members’ roles and responsibilities, as well as their use of research to inform state implementation decisions.

Findings Related to Roles and Activities

CSSS roles. The survey asked respondents about the roles they have assumed within CSSS. In response to these items, CSSS members most often reported serving as conference presenters or participants, at both their state science conferences and the CSSS annual meeting. As compared to other activities, they reported less frequent engagement in CSSS leadership activities, such as committee and board meetings. Overall, associate, honorary, and affiliate members were more often engaged in out-of-state or national activities than state members.

CSSS activities. Respondents were asked to report how often they participated in a range of CSSS activities over the last three years. Overall, respondents reported frequently accessing information from the CSSS listserv, as well as participating in CSSS-sponsored webinars, consulting with CSSS members, and collaborating with other states. State members were significantly less likely to report visiting other states and presenting at national meetings than associate, honorary, and affiliate members. This difference may be related to variation in the roles and responsibilities of state and non-state members.

State activities. Respondents also reported engagement in a variety of state-level activities in the areas of curriculum, instruction, assessment, and professional development. They reported being highly involved in state policy decisions related to science standards, yet they were less involved with the implementation of curricula that reflected those standards. They also reported playing key roles in assessment and professional development, although few respondents had authority to make decisions related to resource allocation or contract selection for assessment systems or professional development providers.

Professional development. Based on a request from CSSS leaders, the 2017 survey included items related to teacher professional development (PD). Respondents were asked to identify the PD offered in their state that afforded teachers the best opportunity to learn about the Framework. Findings revealed that these opportunities tended to be led by a state agency or local leader and funded by federal grant programs. These PD opportunities covered foundational concepts such as the three dimensions of science learning in the Framework, Next Generation Science Standards (NGSS), or instructional planning; however, they tended to pay less attention to designing three-dimensional assessments and developing students’ skills.

Findings Related to CSSS Member Research Use

Specific pieces of research found useful or shared with others. The survey asked respondents to name a specific piece of research they found useful for informing their state’s decisions related to implementation of the Framework, as well as a piece of research they shared with district or school leaders. The research respondents used to support implementation of the Framework most often focused on student learning and classroom assessment, while the research they shared with local leaders most often focused on classroom assessment and pedagogical practices. Few respondents named research focused on the needs or assets of particular student subgroups (e.g., by race/ethnicity, gender, socioeconomic status, or language). Overall, the pieces of research named were primarily research reports or policy briefs, particularly those published by the National Academies of Science, Engineering, and Medicine, as well as peer-reviewed journal articles.

Research trustworthiness. Respondents were asked to indicate why they found the particular pieces of research they named trustworthy. The most often cited reasons were if the research findings applied to their state context, or if the research gave them new ideas to support implementation of the Framework. A less commonly cited reason was that the research methods were rigorous.

Sources used to obtain research. The survey asked respondents how often they obtained research from a list of 13 sources. Of these, many respondents indicated obtaining research through CSSS or colleagues in their state departments of education. Far fewer respondents indicated seeking out research from the National Science Education Leadership Association (NSELA) or from the What Works Clearinghouse (WWC).

Efforts to acquire research. We asked respondents to indicate whether they would seek out research under different conditions. Although a majority said they would look for research to inform a new problem or decision, fewer said they would contact researchers directly under these circumstances, especially researchers they did not already know.

Findings Related to Research Networks

Research networks. The survey asked CSSS members to whom they have turned for research to inform their state’s implementation of the Framework. Findings from these social network questions revealed that associate, honorary, and affiliate members served as prominent sources of research in the areas of curriculum, instruction, assessment, and professional development. University faculty not affiliated with CSSS were also frequently named as sources of research. Additionally, new state members were more likely than veteran state members to name individuals from their own states as sources of research. And, while some respondents facilitated the exchange of research within the professional association, or between CSSS members in different states, others facilitated the exchange of research between CSSS members and researchers unaffiliated with the association.

Relations between activities and networks. Our Year 1 report indicated that an important next step in this study was to explore relationships between CSSS members’ roles and activities and their use of research. We used Year 2 data to examine these relationships, and found that participation in structured CSSS activities, particularly substantive meetings as compared to planning meetings or more informal interactions such as webinars, were important for facilitating the exchange of research among CSSS members.

For a related webinar hosted by NCRPP with leaders from IES-funded researcher-practitioner partnerships, click here.

This final report presents the results of a two-phase study of the Researcher–Practitioner Partnerships in Education Research program, a grant program funded by the Institute of Education Sciences (IES) at the U.S. Department of Education that supports exploratory research within a partnership context. In these partnerships, researchers collaborate with practitioners from at least one state or local education agency on a research project to investigate a problem of practice and to identify strategies to address the key issues.

The descriptive study focused on the first three cohorts of researcher–practitioner partnerships (RPPs) using a mixed-method, cross-case design. It relied on three sources of data: surveys, interviews, and original grant applications. A total of 106 RPP members participated in Phase I, and 114 RPP members participated in Phase II, representing 27 of the 28 funded RPPs in each year.

Progress on Goals

• The RPPs pursued a range of goals in their work together. Partnerships reported they were closest to accomplishing goals related to building a foundation of work together, followed by developing a deep understanding of the focal problem, researchers’ capacity to work in partnership, and a deep understanding of how researchers and practitioners can work together.
• Participants reported making progress on almost all goals, including those related to developing findings that apply to other organizations and improving students’ socio-emotional/non-cognitive outcomes.

Perceived Benefits of Participating in a Partnership

• Researchers and practitioners alike highly valued their participation in partnership work, with almost all of those surveyed either agreeing or strongly agreeing that they would participate in another RPP in the future.
• RPP members reported that partnerships provided local policymakers with new ideas or frameworks or supported the design of professional development, programs, or practices.
• Both researchers and practitioners were involved in the research process, including data collection, analysis, findings dissemination, and communication with stakeholders.

Shifts in Researchers’ and Practitioners’ Engagement with Research and Practice

Note: For a related EdWeek blog post about these shifts, click here.

• Participants described practitioners’ increased appreciation for the value of research, their openness to participating in and using research, and their expanded skills related to conducting and disseminating research.
• Researchers reported expanded understandings about practitioners’ contexts, the value of their input in the research process, and the skills needed to adapt research methods and timely reports of findings to practitioners’ needs.
• Both groups noted they had improved their skills in communicating with stakeholders.

Practitioners’ Use of Research

• Practitioners reported using research to make decisions (instrumental use), to inform their thinking (conceptual use), to persuade others to a point of view (symbolic use), and to integrate research processes into their own work (process use). Practitioners in research roles were significantly more likely to report higher levels of process use of research than their peers in non-research roles.
• Among the activities research evidence might inform, practitioners were most likely to be involved in directing resources to a program, scaling up a program, or designing professional development. When these activities occurred, practitioners reported that about half of their RPP research partners were involved in designing professional development or directing resources to a program.

Useful Pieces of Research

• When asked to name a piece of research that was useful to them, practitioners most frequently named pieces of research that focused on student learning and school organization but that did not have a disciplinary content focus.
• Respondents most often noted that the piece of research they had named was useful because it helped with the design of programs, policies, and initiatives.

Nature of Relationships Prior to the IES Grant

• The majority of partnerships that received IES funding between 2013 and 2015 had participants who had worked together before receiving IES RPP funding. Many had some infrastructure already in place, such as formal data-sharing agreements, broader research agendas beyond the focus of the IES grant, memoranda of understanding (MOUs), and decision-making boards.

Conditions for Starting and Maintaining a Partnership

• The top two conditions for launching an RPP were mutual organizational interest and trust among RPP members. Other conditions included a data-sharing agreement or MOU, individual expertise of RPP members, and organizational leadership.
• Holding regular meetings, mutual organizational interest, and trust among RPP members were top conditions for maintaining a partnership.

Promising Strategies for Overcoming Challenges

• Practitioners reported three main challenges in their partnerships: (1) turnover of those involved in the partnership and within educational organizations more generally; (2) differences in researchers’ and practitioners’ timelines or pace of work; and (3) having the “right people at the table” in terms of active members in the partnership with decision-making authority to act on the partnerships’ findings.
• Participants shared strategies they felt were useful in navigating these challenges, including building strong, trusting relationships, communicating regularly, and being flexible enough to adjust course as needed.

Organizational Conditions in the Practice Organization

• The majority of practitioners agreed that research was seen as a useful source of information in their organization. However, only half of practitioners reported having enough time, space, and resources to make sense of new information from their partners or that new knowledge was regularly communicated across departments.
• A majority of practitioners reported that it was easy to see the connections between their organizations’ initiatives and work with external partners, yet two-thirds of practitioners reported that organizational leadership did not coordinate work so as to limit conflicts or reduce overlap between their organizations’ initiatives and partnership work.

Plans for Ongoing Work Together

• The majority of partnerships had continued working together past the end of the IES RPP grant or planned to continue to do so.
• Six of the 27 partnerships had successfully applied for and received additional funding; another five applied for additional funding but did not receive it. Ten ongoing partnerships had plans to apply for additional funding, while the remaining six did not have plans to apply for additional funding at the time of the Phase II survey.

For a related webinar hosted by NCRPP with leaders from IES-funded researcher-practitioner partnerships, click here.

This report presents the results of a descriptive study of the Researcher–Practitioner Partnerships in Education Research program, a two-year grant program funded by the Institute of Education Sciences at the U.S. Department of Education that supports exploratory research within a partnership context. In these partnerships, researchers collaborate with practitioners from at least one state or local education agency on a research project to investigate a problem of practice and to identify strategies to address the key issues.

The descriptive study focused on the first three cohorts of researcher–practitioner partnerships (RPPs) using a mixed-method, cross-case design. It relied on three sources of data: surveys, interviews, and original grant applications. A total of 106 participants from 27 of the 28 funded RPPs completed the survey, the interview, or both.

Goals

• The most commonly reported goals of the RPPs were ones emphasized within the request for applications (RFA) for the program: conducting and using research, and impacting local improvement efforts. RPPs also pursued the goals of cultivating partnership relationships, increasing the capacity of researchers and practitioners to engage in partnership work, and informing the work of others through the sharing of strategies and tools with people outside the partnership.
• Partnerships reported the greatest progress in their initial work toward the goals of building relationships and refining their understandings of problems the partnership could address. They reported somewhat less progress for longer-term goals, such as improving organizational policies and processes in the educational organization and impacting students’ outcomes.

Conducting and Using Research

• The majority of partnerships focused on descriptive or exploratory questions, and some focused on investigating causal relationships or validating measures or constructs.
• Most proposed to use mixed methods and to draw on both existing and new data sources. To carry out these research plans, partnerships created multi-sector datasets, engaged in secondary analyses of student-, teacher-, and school-level data, conducted interviews and focus groups, created case study reports, and more.

Activities and Communication

• All partnerships were engaged in problem refinement and analysis of data, and nearly half were doing some design work together that involved developing programs and practices that they were testing or planning to test.
• Participants in the RPPs maintained regular and frequent communication across partners, with most members engaging in weekly emails or phone calls.

Challenges

• Roughly half of the partnerships identified organizational turnover and obtaining usable data as key challenges to their work together.
• Other leading challenges were synchronizing schedules of researchers and practitioners and accommodating different timelines for getting work done.

Perceptions of Partnership

• Researchers and practitioners alike highly valued their participation in partnership work, with 100% of those surveyed either agreeing or strongly agreeing that they would participate in an RPP in the future.
• Practitioners named several benefits, including helping to shift organizational culture for research use and increasing access to resources and expertise aimed at understanding and addressing a specific problem of practice.
• Both researchers and practitioners also felt that the quality and applicability of research increased as a result of the partnership.

This report presents the findings of our study of research use among state science leaders, led by Megan Hopkins, assistant professor of Education Studies at the University of California, San Diego.

We surveyed members of the Council of State Science Supervisors (CSSS), a professional organization composed of science education specialists who serve at the state, territorial, or the protectorate educational agency in the United States and U.S. territories. Its members include all current supervisors of science education in State Education Agencies, as well as former supervisors. Within their jurisdictions, supervisors play key roles in directing efforts at ensuring excellence and equity in science education as outlined in A Framework for K-12 Science Education, which guides the development of science standards in many states and was used in the articulation of the Next Generation Science Standards.

The study found that networks of policy makers and educational leaders facilitate access to research that informs decisions related to statewide science education reform.

We asked CSSS members a variety of questions to understand the activities they engage in as state science supervisors, as well as how they access and use research in their work. Sixty-one CSSS members across 38 states responded to our survey and reported that their work most frequently involves activities that attend to key components of their state’s education systems, including: reviewing and developing state science standards, designing state science assessments, and designing or conducting teacher professional development.

The research that CSSS members used to inform their state’s decisions focused on instructional practices and student learning in science, such as the Ready, Set, Science! report by the National Academies Press, and the edited volume, Working with Big Ideas. Very little of the research that CSSS members used, however, focused on science education reform in diverse contexts.

CSSS members most often accessed research through their professional networks, including through CSSS, the National Science Teachers Association (NSTA), and colleagues in state departments of education, rather than via other research outlets such as the What Works Clearinghouse. Broad engagement in CSSS networks, such as through CSSS-sponsored meetings and workshops, appeared to facilitate the use of research more than activities like visiting or collaborating with other states.

These professional opportunities likely facilitated the development of a robust research network for CSSS members, who reported turning to a variety of individuals for research to inform their state’s decisions related to the Framework. These individuals represented a range of organizational affiliations, and included 16 current CSSS members, 7 former CSSS members, 8 state department of education leaders, 7 local educational leaders, 29 individuals working for educational non-profit organizations, and as many as 75 university researchers.

CSSS members had a well-connected research network, indicating that most individuals could access one another via other individuals in the network, such that research could potentially flow across the network relatively easily. A few former CSSS members and university researchers were prominent in CSSS members’ research network, suggesting that the research they shared was particularly influential. Former CSSS members also were key sources of research, suggesting that their participation in CSSS activities helped to preserve association memory around the kinds of research that inform members’ work, which is important for sustaining implementation efforts.

The instrument used to survey a nationally representative sample of school and district leaders, called the Survey of Practitioners’ Use of Research (SPUR), is now available for use by those interested in research use.

Visit the SPUR download page to read the Terms of Use and access the download.

The NCRPP research use survey focuses on the core of “research use” in three related, but separate, constructs: the ways that school and district leaders 1) apply research in their decision-making processes; 2) value research as an important component for decision-making; and 3) evaluate research quality. The survey instrument also includes measure of organizational culture for research use and open-ended reports of research that educational leaders find useful.

The survey can help researchers describe and compare the prevalence of different kinds of research use among district and school decision makers and be useful for evaluations of actions intended to improve research use.

An extensive survey development process to establish the validity and reliability of measures. Two different groups of advisors with practical and research expertise in the use of research reviewed items. The team conducted cognitive interviews with 40 educational leaders to ensure items were comprehensible and to determine whether items elicited the focal constructs. We assessed the internal consistency of scales used and used item response theory to evaluate scales’ ability to discriminate amongst different respondents. Overall, all of the scales showed good reliability. Information about the survey development process, reliability of scales, and findings can be found in Technical Report #1 (see below).

This technical report presents results of a nationally representative survey of principals and district leaders in the nation’s mid-sized and large school districts conducted by the National Center for Research in Policy and Practice. A total of 733 school and district leaders responded to the survey from 45 states and 485 school districts.

The survey focused on how these educational leaders use research to inform their decision making, identifying three types of research use defined in earlier studies: instrumental use of research to guide or inform a specific decision; conceptual use of research to inform the way a person views a problem or possible solutions; and symbolic use of research to persuade others or legitimate a decision already made.

School and district leaders also answered questions about their attitudes toward research, their efforts to acquire it, and the culture of research use in their organizations. Finally, an open-ended survey item asked school and district leaders to name a specific piece of research they found useful.

Types of Use

• Respondents most commonly reported instrumental use of research. Common forms of instrumental use were to design professional development for teachers and administrators and direct resources to programs.
• With respect to conceptual use, 71% of respondents indicated that the research they encountered had expanded their understanding of an issue.
• Among the symbolic uses presented to respondents, leaders most frequently reported using research to convince others or using research selectively to support a decision.

Sources Leaders Used to Obtain Research

• Leaders most frequently accessed research through professional associations, professional conferences, and colleagues.
• Leaders less frequently accessed research directly through individual researchers or from the What Works Clearinghouse, the National Center for Education Statistics, or the Regional Educational Laboratories. These sources overlap with professional associations, conferences, and colleagues, such that these findings indicate points of access, rather than relative influence.

Attitudes Toward Research

• Leaders endorsed the idea that research is relevant to practice, but indicated that the time lag between conducting and publishing research can decrease its usefulness.
• Respondents reported very positive attitudes about the value of educational research, with nearly all endorsing the ideas that research can address practical problems and that researchers provide a valuable service to educational practitioners.
• Leaders were mixed in their perceptions of the credibility of research, with half saying that researchers could be biased and about a third saying that researchers often framed their results to make a particular point.

Effort to Acquire Research

• Although a majority of leaders said they would look for research to inform a new problem or decision, few said they would contact researchers directly.

Knowledge of How to Interpret Conclusions from Research

• The majority of responses indicated understandings of the role of purposeful sampling in qualitative research and how to interpret effect sizes in quantitative research.
• Few leaders drew accurate conclusions about what can be learned from a case study and half of respondents did not respond accurately to a question about random assignment.

Culture of Research Use

• Although most leaders reported that research is viewed as a useful source of information in their district or department, a majority disagreed with the statement that people expected claims made in meetings to be backed up by research.

Specific Pieces of Research Leaders Found Useful

• The pieces of research that school and district leaders named as useful were most often books, research or policy reports, or peer-reviewed journal articles.
• Most named research focused on instructional practices and learning in the classroom, though few mentioned research pertaining to specific subject matter content areas.
• Respondents most often claimed to use research named to support their own learning, inform the design of programs, and provide instructional leadership.

In spring 2015, 271 central office staff members from 32 of the largest districts in the U.S. completed a survey about their attitudes toward research and how they use research in their work.

This report shares the results of that survey overall and by four types of professional roles in the district: Area & School Supervision; Curriculum & Instruction; Special Education; and Assessment, Accountability & Research.

• District leaders viewed research as valuable, credible, and relevant.
  The overwhelming majority reported that education research was valuable (95%), credible (85%), and relevant to their work (87%). However, participants were split regarding whether there was a “disconnect” between the worlds of research and practice.
• District leaders used research to inform decisions, expand understanding, and persuade others.
  Almost all reported using research to inform decisions (99%). In addition, all participants reported using research to inform their thinking or expand their understanding of an issue (100%), and almost all reported using research to persuade or convince others of their position (96%).
• The pieces of research that district leaders named as useful were most often books, research or policy reports, or journal articles focused on instructional practices.
  When participants were asked to name a piece of research that had been useful in the past year, they most frequently named books (36%), followed by research or policy reports (29%) and peer-reviewed journal articles (23%). Most pieces focused on instructional practices, but did not focus on a particular content area or subgroup of students. Similar to the various types of research use reported, participants most often claimed that the pieces they named were useful for developing their own professional learning and knowledge (29%), followed by providing instructional leadership for others (25%) and designing policies and programs (20%).