Findings from a Survey of State Science Leaders: Year 2
This report presents results from a study of the Council of State Science Supervisors (CSSS). CSSS is a professional association for state science leaders that works to sustain and nurture a dynamic learning community and to empower its members to be effective and articulate advocates for quality science education at the local, state, and national levels.
Given that many state science supervisors are working to support implementation of the Framework for K-12 Science Education, we sought to develop an understanding of CSSS members’ roles and responsibilities, as well as their use of research to inform state implementation decisions.
Findings Related to Roles and Activities
CSSS roles. The survey asked respondents about the roles they have assumed within CSSS. In response to these items, CSSS members most often reported serving as conference presenters or participants, at both their state science conferences and the CSSS annual meeting. As compared to other activities, they reported less frequent engagement in CSSS leadership activities, such as committee and board meetings. Overall, associate, honorary, and affiliate members were more often engaged in out-of-state or national activities than state members.
CSSS activities. Respondents were asked to report how often they participated in a range of CSSS activities over the last three years. Overall, respondents reported frequently accessing information from the CSSS listserv, as well as participating in CSSS-sponsored webinars, consulting with CSSS members, and collaborating with other states. State members were significantly less likely to report visiting other states and presenting at national meetings than associate, honorary, and affiliate members. This difference may be related to variation in the roles and responsibilities of state and non-state members.
State activities. Respondents also reported engagement in a variety of state-level activities in the areas of curriculum, instruction, assessment, and professional development. They reported being highly involved in state policy decisions related to science standards, yet they were less involved with the implementation of curricula that reflected those standards. They also reported playing key roles in assessment and professional development, although few respondents had authority to make decisions related to resource allocation or contract selection for assessment systems or professional development providers.
Professional development. Based on a request from CSSS leaders, the 2017 survey included items related to teacher professional development (PD). Respondents were asked to identify the PD offered in their state that afforded teachers the best opportunity to learn about the Framework. Findings revealed that these opportunities tended to be led by a state agency or local leader and funded by federal grant programs. These PD opportunities covered foundational concepts such as the three dimensions of science learning in the Framework, Next Generation Science Standards (NGSS), or instructional planning; however, they tended to pay less attention to designing three-dimensional assessments and developing students’ skills.
Findings Related to CSSS Member Research Use
Specific pieces of research found useful or shared with others. The survey asked respondents to name a specific piece of research they found useful for informing their state’s decisions related to implementation of the Framework, as well as a piece of research they shared with district or school leaders. The research respondents used to support implementation of the Framework most often focused on student learning and classroom assessment, while the research they shared with local leaders most often focused on classroom assessment and pedagogical practices. Few respondents named research focused on the needs or assets of particular student subgroups (e.g., by race/ethnicity, gender, socioeconomic status, or language). Overall, the pieces of research named were primarily research reports or policy briefs, particularly those published by the National Academies of Science, Engineering, and Medicine, as well as peer-reviewed journal articles.
Research trustworthiness. Respondents were asked to indicate why they found the particular pieces of research they named trustworthy. The most often cited reasons were if the research findings applied to their state context, or if the research gave them new ideas to support implementation of the Framework. A less commonly cited reason was that the research methods were rigorous.
Sources used to obtain research. The survey asked respondents how often they obtained research from a list of 13 sources. Of these, many respondents indicated obtaining research through CSSS or colleagues in their state departments of education. Far fewer respondents indicated seeking out research from the National Science Education Leadership Association (NSELA) or from the What Works Clearinghouse (WWC).
Efforts to acquire research. We asked respondents to indicate whether they would seek out research under different conditions. Although a majority said they would look for research to inform a new problem or decision, fewer said they would contact researchers directly under these circumstances, especially researchers they did not already know.
Findings Related to Research Networks
Research networks. The survey asked CSSS members to whom they have turned for research to inform their state’s implementation of the Framework. Findings from these social network questions revealed that associate, honorary, and affiliate members served as prominent sources of research in the areas of curriculum, instruction, assessment, and professional development. University faculty not affiliated with CSSS were also frequently named as sources of research. Additionally, new state members were more likely than veteran state members to name individuals from their own states as sources of research. And, while some respondents facilitated the exchange of research within the professional association, or between CSSS members in different states, others facilitated the exchange of research between CSSS members and researchers unaffiliated with the association.
Relations between activities and networks. Our Year 1 report indicated that an important next step in this study was to explore relationships between CSSS members’ roles and activities and their use of research. We used Year 2 data to examine these relationships, and found that participation in structured CSSS activities, particularly substantive meetings as compared to planning meetings or more informal interactions such as webinars, were important for facilitating the exchange of research among CSSS members.
Click the image above to download the full report and read more about findings and recommendations.
A Descriptive Study of the IES Researcher–Practitioner Partnerships in Education Research Program: Final Report
For a related webinar hosted by NCRPP with leaders from IES-funded researcher-practitioner partnerships, click here.
This final report presents the results of a two-phase study of the Researcher–Practitioner Partnerships in Education Research program, a grant program funded by the Institute of Education Sciences (IES) at the U.S. Department of Education that supports exploratory research within a partnership context. In these partnerships, researchers collaborate with practitioners from at least one state or local education agency on a research project to investigate a problem of practice and to identify strategies to address the key issues.
The descriptive study focused on the first three cohorts of researcher–practitioner partnerships (RPPs) using a mixed-method, cross-case design. It relied on three sources of data: surveys, interviews, and original grant applications. A total of 106 RPP members participated in Phase I, and 114 RPP members participated in Phase II, representing 27 of the 28 funded RPPs in each year.
Progress on Goals
- The RPPs pursued a range of goals in their work together. Partnerships reported they were closest to accomplishing goals related to building a foundation of work together, followed by developing a deep understanding of the focal problem, researchers’ capacity to work in partnership, and a deep understanding of how researchers and practitioners can work together.
- Participants reported making progress on almost all goals, including those related to developing findings that apply to other organizations and improving students’ socio-emotional/non-cognitive outcomes.
Perceived Benefits of Participating in a Partnership
- Researchers and practitioners alike highly valued their participation in partnership work, with almost all of those surveyed either agreeing or strongly agreeing that they would participate in another RPP in the future.
- RPP members reported that partnerships provided local policymakers with new ideas or frameworks or supported the design of professional development, programs, or practices.
- Both researchers and practitioners were involved in the research process, including data collection, analysis, findings dissemination, and communication with stakeholders.
Shifts in Researchers’ and Practitioners’ Engagement with Research and Practice
Note: For a related EdWeek blog post about these shifts, click here.
- Participants described practitioners’ increased appreciation for the value of research, their openness to participating in and using research, and their expanded skills related to conducting and disseminating research.
- Researchers reported expanded understandings about practitioners’ contexts, the value of their input in the research process, and the skills needed to adapt research methods and timely reports of findings to practitioners’ needs.
- Both groups noted they had improved their skills in communicating with stakeholders.
Practitioners’ Use of Research
- Practitioners reported using research to make decisions (instrumental use), to inform their thinking (conceptual use), to persuade others to a point of view (symbolic use), and to integrate research processes into their own work (process use). Practitioners in research roles were significantly more likely to report higher levels of process use of research than their peers in non-research roles.
- Among the activities research evidence might inform, practitioners were most likely to be involved in directing resources to a program, scaling up a program, or designing professional development. When these activities occurred, practitioners reported that about half of their RPP research partners were involved in designing professional development or directing resources to a program.
Useful Pieces of Research
- When asked to name a piece of research that was useful to them, practitioners most frequently named pieces of research that focused on student learning and school organization but that did not have a disciplinary content focus.
- Respondents most often noted that the piece of research they had named was useful because it helped with the design of programs, policies, and initiatives.
Nature of Relationships Prior to the IES Grant
- The majority of partnerships that received IES funding between 2013 and 2015 had participants who had worked together before receiving IES RPP funding. Many had some infrastructure already in place, such as formal data-sharing agreements, broader research agendas beyond the focus of the IES grant, memoranda of understanding (MOUs), and decision-making boards.
Conditions for Starting and Maintaining a Partnership
- The top two conditions for launching an RPP were mutual organizational interest and trust among RPP members. Other conditions included a data-sharing agreement or MOU, individual expertise of RPP members, and organizational leadership.
- Holding regular meetings, mutual organizational interest, and trust among RPP members were top conditions for maintaining a partnership.
Promising Strategies for Overcoming Challenges
- Practitioners reported three main challenges in their partnerships: (1) turnover of those involved in the partnership and within educational organizations more generally; (2) differences in researchers’ and practitioners’ timelines or pace of work; and (3) having the “right people at the table” in terms of active members in the partnership with decision-making authority to act on the partnerships’ findings.
- Participants shared strategies they felt were useful in navigating these challenges, including building strong, trusting relationships, communicating regularly, and being flexible enough to adjust course as needed.
Organizational Conditions in the Practice Organization
- The majority of practitioners agreed that research was seen as a useful source of information in their organization. However, only half of practitioners reported having enough time, space, and resources to make sense of new information from their partners or that new knowledge was regularly communicated across departments.
- A majority of practitioners reported that it was easy to see the connections between their organizations’ initiatives and work with external partners, yet two-thirds of practitioners reported that organizational leadership did not coordinate work so as to limit conflicts or reduce overlap between their organizations’ initiatives and partnership work.
Plans for Ongoing Work Together
- The majority of partnerships had continued working together past the end of the IES RPP grant or planned to continue to do so.
- Six of the 27 partnerships had successfully applied for and received additional funding; another five applied for additional funding but did not receive it. Ten ongoing partnerships had plans to apply for additional funding, while the remaining six did not have plans to apply for additional funding at the time of the Phase II survey.
Click the image above to download the full report and read more about findings and recommendations.
Second Round of Online Course Starting February 5th: Using Research to Inform Decisions
Hosted by the Harvard Graduate School of Education, NCRPP is offering an online course February 5-18, 2018 that will help district leaders, school leaders, and state education officials learn how to find, evaluate, and use research to inform the decisions they make.
Learn from the co-principal investigators of NCRPP who are leading scholars in the field:
- Heather Hill, Jerome T. Murphy Professor in Education at the Harvard Graduate School of Education
- Bill Penuel, Professor of Educational Psychology and Learning Sciences at the University of Colorado Boulder
- Derek Briggs, Professor of Research and Evaluation Methodology at the University of Colorado Boulder
- Cynthia Coburn, Professor at the School of Education and Social Policy at Northwestern University
How to Register
- Dates: February 5-18, 2018
- Cost: $149 per person
- Register: gse.harvard.edu/ppe/rid
Federal guidelines now ask school and district staff to use research evidence when making decisions about policies and programs. Despite these realities, finding and using solid research can be a difficult task.
Using Research to Inform Decisions will guide you in how to find and evaluate pertinent research, with a constant eye toward how such research can be used to inform various school- and district-level decisions. The workshop provides an overview of how practitioners use research and offers several simple tools to help you find the study you need. You will also learn strategies and structures that help foster an environment where research truly informs decision-making.
This two-week online workshop is jointly led by HGSE Professor Heather Hill, University of Colorado Professor Bill Penuel, and other leaders in the field. Video lectures, readings, facilitated online discussions, and exploration exercises will help you engage with research and gain confidence in using evidence to make sound decisions.
By completing the program, you will be able to:
- Explore the nature of research use and current national patterns in research use
- Learn how to find relevant research and read it accurately
- Evaluate research design, and be able to judge the credibility of the research
- Implement strategies for increasing the use of evidence within your own educational setting
The online instruction, discussions, and job-embedded practice should take five to seven hours over the duration of the program. Participants who complete all individual assessments and contribute to group discussions will receive a certificate indicating completion of five clock hours of instruction.
*Note: all content is self-paced, including group discussions. Facilitators will moderate discussion boards throughout the two-week program.
A Descriptive Study of the IES Researcher-Practitioner Partnerships Program: Interim Report
For a related webinar hosted by NCRPP with leaders from IES-funded researcher-practitioner partnerships, click here.
This report presents the results of a descriptive study of the Researcher–Practitioner Partnerships in Education Research program, a two-year grant program funded by the Institute of Education Sciences at the U.S. Department of Education that supports exploratory research within a partnership context. In these partnerships, researchers collaborate with practitioners from at least one state or local education agency on a research project to investigate a problem of practice and to identify strategies to address the key issues.
The descriptive study focused on the first three cohorts of researcher–practitioner partnerships (RPPs) using a mixed-method, cross-case design. It relied on three sources of data: surveys, interviews, and original grant applications. A total of 106 participants from 27 of the 28 funded RPPs completed the survey, the interview, or both.
The most commonly reported goals of the RPPs were ones emphasized within the request for applications (RFA) for the program: conducting and using research, and impacting local improvement efforts. RPPs also pursued the goals of cultivating partnership relationships, increasing the capacity of researchers and practitioners to engage in partnership work, and informing the work of others through the sharing of strategies and tools with people outside the partnership.
Partnerships reported the greatest progress in their initial work toward the goals of building relationships and refining their understandings of problems the partnership could address. They reported somewhat less progress for longer-term goals, such as improving organizational policies and processes in the educational organization and impacting students’ outcomes.
Conducting and Using Research
The majority of partnerships focused on descriptive or exploratory questions, and some focused on investigating causal relationships or validating measures or constructs.
Most proposed to use mixed methods and to draw on both existing and new data sources. To carry out these research plans, partnerships created multi-sector datasets, engaged in secondary analyses of student-, teacher-, and school-level data, conducted interviews and focus groups, created case study reports, and more.
Activities and Communication
All partnerships were engaged in problem refinement and analysis of data, and nearly half were doing some design work together that involved developing programs and practices that they were testing or planning to test.
Participants in the RPPs maintained regular and frequent communication across partners, with most members engaging in weekly emails or phone calls.
Roughly half of the partnerships identified organizational turnover and obtaining usable data as key challenges to their work together.
Other leading challenges were synchronizing schedules of researchers and practitioners and accommodating different timelines for getting work done.
Perceptions of Partnership
Researchers and practitioners alike highly valued their participation in partnership work, with 100% of those surveyed either agreeing or strongly agreeing that they would participate in an RPP in the future.
Practitioners named several benefits, including helping to shift organizational culture for research use and increasing access to resources and expertise aimed at understanding and addressing a specific problem of practice.
Both researchers and practitioners also felt that the quality and applicability of research increased as a result of the partnership.
Findings from a Survey of State Science Leaders
This report presents the findings of our study of research use among state science leaders, led by Megan Hopkins, assistant professor of Education Studies at the University of California, San Diego.
We surveyed members of the Council of State Science Supervisors (CSSS), a professional organization composed of science education specialists who serve at the state, territorial, or the protectorate educational agency in the United States and U.S. territories. Its members include all current supervisors of science education in State Education Agencies, as well as former supervisors. Within their jurisdictions, supervisors play key roles in directing efforts at ensuring excellence and equity in science education as outlined in A Framework for K-12 Science Education, which guides the development of science standards in many states and was used in the articulation of the Next Generation Science Standards.
The study found that networks of policy makers and educational leaders facilitate access to research that informs decisions related to statewide science education reform.
USING RESEARCH IN IMPLEMENTATION EFFORTS
We asked CSSS members a variety of questions to understand the activities they engage in as state science supervisors, as well as how they access and use research in their work. Sixty-one CSSS members across 38 states responded to our survey and reported that their work most frequently involves activities that attend to key components of their state’s education systems, including: reviewing and developing state science standards, designing state science assessments, and designing or conducting teacher professional development.
The research that CSSS members used to inform their state’s decisions focused on instructional practices and student learning in science, such as the Ready, Set, Science! report by the National Academies Press, and the edited volume, Working with Big Ideas. Very little of the research that CSSS members used, however, focused on science education reform in diverse contexts.
LEVERAGING NETWORKS FOR RESEARCH USE
CSSS members most often accessed research through their professional networks, including through CSSS, the National Science Teachers Association (NSTA), and colleagues in state departments of education, rather than via other research outlets such as the What Works Clearinghouse. Broad engagement in CSSS networks, such as through CSSS-sponsored meetings and workshops, appeared to facilitate the use of research more than activities like visiting or collaborating with other states.
These professional opportunities likely facilitated the development of a robust research network for CSSS members, who reported turning to a variety of individuals for research to inform their state’s decisions related to the Framework. These individuals represented a range of organizational affiliations, and included 16 current CSSS members, 7 former CSSS members, 8 state department of education leaders, 7 local educational leaders, 29 individuals working for educational non-profit organizations, and as many as 75 university researchers.
CSSS MEMBERS’ RESEARCH NETWORK
CSSS members had a well-connected research network, indicating that most individuals could access one another via other individuals in the network, such that research could potentially flow across the network relatively easily. A few former CSSS members and university researchers were prominent in CSSS members’ research network, suggesting that the research they shared was particularly influential. Former CSSS members also were key sources of research, suggesting that their participation in CSSS activities helped to preserve association memory around the kinds of research that inform members’ work, which is important for sustaining implementation efforts.
Survey of Practitioners’ Use of Research (SPUR)
The instrument used to survey a nationally representative sample of school and district leaders, called the Survey of Practitioners’ Use of Research (SPUR), is now available for use by those interested in research use.
The NCRPP research use survey focuses on the core of “research use” in three related, but separate, constructs: the ways that school and district leaders 1) apply research in their decision-making processes; 2) value research as an important component for decision-making; and 3) evaluate research quality. The survey instrument also includes measure of organizational culture for research use and open-ended reports of research that educational leaders find useful.
The survey can help researchers describe and compare the prevalence of different kinds of research use among district and school decision makers and be useful for evaluations of actions intended to improve research use.
An extensive survey development process to establish the validity and reliability of measures. Two different groups of advisors with practical and research expertise in the use of research reviewed items. The team conducted cognitive interviews with 40 educational leaders to ensure items were comprehensible and to determine whether items elicited the focal constructs. We assessed the internal consistency of scales used and used item response theory to evaluate scales’ ability to discriminate amongst different respondents. Overall, all of the scales showed good reliability. Information about the survey development process, reliability of scales, and findings can be found in Technical Report #1 (see below).
Findings from a National Study on Research Use Among School and District Leaders
This technical report presents results of a nationally representative survey of principals and district leaders in the nation’s mid-sized and large school districts conducted by the National Center for Research in Policy and Practice. A total of 733 school and district leaders responded to the survey from 45 states and 485 school districts.
The survey focused on how these educational leaders use research to inform their decision making, identifying three types of research use defined in earlier studies: instrumental use of research to guide or inform a specific decision; conceptual use of research to inform the way a person views a problem or possible solutions; and symbolic use of research to persuade others or legitimate a decision already made.
School and district leaders also answered questions about their attitudes toward research, their efforts to acquire it, and the culture of research use in their organizations. Finally, an open-ended survey item asked school and district leaders to name a specific piece of research they found useful.
Types of Use
- Respondents most commonly reported instrumental use of research. Common forms of instrumental use were to design professional development for teachers and administrators and direct resources to programs.
- With respect to conceptual use, 71% of respondents indicated that the research they encountered had expanded their understanding of an issue.
- Among the symbolic uses presented to respondents, leaders most frequently reported using research to convince others or using research selectively to support a decision.
Sources Leaders Used to Obtain Research
- Leaders most frequently accessed research through professional associations, professional conferences, and colleagues.
- Leaders less frequently accessed research directly through individual researchers or from the What Works Clearinghouse, the National Center for Education Statistics, or the Regional Educational Laboratories. These sources overlap with professional associations, conferences, and colleagues, such that these findings indicate points of access, rather than relative influence.
Attitudes Toward Research
- Leaders endorsed the idea that research is relevant to practice, but indicated that the time lag between conducting and publishing research can decrease its usefulness.
- Respondents reported very positive attitudes about the value of educational research, with nearly all endorsing the ideas that research can address practical problems and that researchers provide a valuable service to educational practitioners.
- Leaders were mixed in their perceptions of the credibility of research, with half saying that researchers could be biased and about a third saying that researchers often framed their results to make a particular point.
Effort to Acquire Research
- Although a majority of leaders said they would look for research to inform a new problem or decision, few said they would contact researchers directly.
Knowledge of How to Interpret Conclusions from Research
- The majority of responses indicated understandings of the role of purposeful sampling in qualitative research and how to interpret effect sizes in quantitative research.
- Few leaders drew accurate conclusions about what can be learned from a case study and half of respondents did not respond accurately to a question about random assignment.
Culture of Research Use
- Although most leaders reported that research is viewed as a useful source of information in their district or department, a majority disagreed with the statement that people expected claims made in meetings to be backed up by research.
Specific Pieces of Research Leaders Found Useful
- The pieces of research that school and district leaders named as useful were most often books, research or policy reports, or peer-reviewed journal articles.
- Most named research focused on instructional practices and learning in the classroom, though few mentioned research pertaining to specific subject matter content areas.
- Respondents most often claimed to use research named to support their own learning, inform the design of programs, and provide instructional leadership.
Findings from a Survey of Research Attitudes and Use in the Nation's Largest Districts
In spring 2015, 271 central office staff members from 32 of the largest districts in the U.S. completed a survey about their attitudes toward research and how they use research in their work.
This report shares the results of that survey overall and by four types of professional roles in the district: Area & School Supervision; Curriculum & Instruction; Special Education; and Assessment, Accountability & Research.
District leaders viewed research as valuable, credible, and relevant.
The overwhelming majority reported that education research was valuable (95%), credible (85%), and relevant to their work (87%). However, participants were split regarding whether there was a “disconnect” between the worlds of research and practice.
District leaders used research to inform decisions, expand understanding, and persuade others.
Almost all reported using research to inform decisions (99%). In addition, all participants reported using research to inform their thinking or expand their understanding of an issue (100%), and almost all reported using research to persuade or convince others of their position (96%).
The pieces of research that district leaders named as useful were most often books, research or policy reports, or journal articles focused on instructional practices.
When participants were asked to name a piece of research that had been useful in the past year, they most frequently named books (36%), followed by research or policy reports (29%) and peer-reviewed journal articles (23%). Most pieces focused on instructional practices, but did not focus on a particular content area or subgroup of students. Similar to the various types of research use reported, participants most often claimed that the pieces they named were useful for developing their own professional learning and knowledge (29%), followed by providing instructional leadership for others (25%) and designing policies and programs (20%).
High-quality educational research that could shed light on effective policies and practices is increasingly accessible to districts and schools. Since its establishment in 2002, the Institute of Education Sciences (IES) at the U.S. Department of Education has funded dozens of field- initiated efficacy and scale-up studies of interventions, released multiple evaluation studies of major policy initiatives, supported rigorous studies of programs through the Regional Educational Laboratories, and funded training grants to prepare new scholars to conduct more scientific research in education. Efforts such as the What Works Clearinghouse, Doing What Works web site, and syntheses of research presented in practice guides, are all efforts to increase access to scientific research.
Providing access is only the first step to improving student outcomes. Districts and schools must then use this research to improve the quality of learning opportunities they provide to students. At present, there is a small body of inquiry on how district and school leaders use research. The results from such studies are consistent: research is rarely used, and not in the linear fashion imagined by most (Coburn, Honig, & Stein, 2009). Research use involves interactive processes, including contention, persuasion, negotiation, and sensemaking (Amara, Ouimet, & Landry, 2004; Contandriopoulos, Lemire, Denis, & Tremblay, 2010; Earl, 1995). It requires leaders to make sense of conclusions, deliberate about their relevance to the current context, and create policies that reflect agreements about what the research suggests they should do in that context.
We need to understand much more about the interactive processes involved in research use, or what Tseng (2007) has called “the demand side” of research use before we can improve district and school leaders’ research use. We need measures of research use to track progress at both the central office and school levels. We also need to understand better how school and district leaders currently use research. It is only by understanding how these local leaders actually make decisions, and the role of research in this process, that we can begin to design interventions that promote more effective uses of research. And, finally, we need more research on existing strategies to promote research use. Today, there are a number of efforts to foster stronger partnerships between researchers and practitioners as a strategy to increase research use (Coburn, Penuel, & Geil, 2013), but little is known about the role of partnership research in decision making.