Most studies of how policy makers and leaders use research in their work rely largely on self-report via surveys or interviews. There are clear limitations with relying on only these approaches. When pressure exists to use research, respondents may be more likely to over-report research use on surveys and interviews. Surveys and interviews elicit retrospective accounts, producing reports smoothed by the passage of time and limitations of memory. They can fail to capture what we know is an interactive process, one that unfolds over time and involves multiple actors across an organization.
To better understand the role of research in school and district decision making, NCRPP is developing a new observation protocol. In this post, we share three key ideas that inform our design of the protocol and discuss some of the key challenges we’re facing as we develop and test this tool.
Key Idea #1: Decisions “Accrete” Over Time, Across Settings
It’s surprisingly hard to identify a single point in time when an educational leader or group “makes a decision.” As Carol Weiss once wrote, decisions accrete. That is, they emerge through a series of conversations and meetings with different groups of people over time and across settings.
Design implications: Because decisions accrete over time, an observation protocol should allow investigators to develop evidence of research use from multiple observations of school or district meetings, not just one. In addition, even when they attend many meetings, outside observers are likely to miss important information that insiders have about what’s going on in those meetings. In the next phase of tool development, we plan to pair observations with interviews, in order to locate a given meeting within a larger “issue trajectory” that district leaders are discussing.
Key Idea #2: Research is One Form of Evidence Among Many
NCRPP is focused primarily how educators engage research evidence in their work, but we know there are other kinds of evidence that figure prominently in educational decision-making and deliberation, like personal anecdotes, data, what’s “typical,” or what’s mandated by other district policies. In district debates and discussions, reasons given may be endorsed, challenged, or enriched for a given decision.
Design implications: As we observe when and how research is brought up, we will attend to other kinds of evidence that people name and find persuasive. We need to track research as part of the “reason giving” at the core of debate in policy contexts. Within the give and take of deliberation, we will focus on how research figures into the reasons that participants give for claims they make about issues districts are facing and the kinds of solutions districts should pursue.
Key Idea #3: Interaction Dynamics Matter
Observational methods are particularly valuable for understanding the interaction dynamics and the flow of talk that happens when people deliberate. The dynamics of meetings shift based on who’s at the table, what’s on the table for debate, the unspoken norms and rules of engagement, and the structures or routines in place. These dynamics can shape the meaning and interpretation of research and the role that it plays in a debate.
Design implications: As we observe meetings, we will identify key dimensions of the setting that could influence people’s interactions. These include the meeting format and setting, the topic of the debate, the participants and their roles, and the function of certain routines or structures.
Challenges and the Future of our Work
We face several design challenges in our work to create a useful observation protocol.
First, we are wrestling with predicting who might use our protocol. We are discussing whether this tool is likely to be valuable to researchers in university and non-profit settings or whether district leaders might be interested in using it, as well.
Second, we’re struck by how much “local knowledge” of the district context, the leaders, and what’s at stake in debates is needed to draw valid inferences about what people are up to and about the significance of research within deliberations. When we watch video recordings of meetings, those of us with deep familiarity with the district context see entirely different things going on than do team members who have not spent time in the district. This has profound implications for what it means to measure research use through an observational protocol.
One key objective of NCRPP is to help the field develop a suite of measures for understanding the role of research within actual school and district settings. Over the next year, we will continue share the choices we make and challenges we face as we engage in this design work. Part of our goal is to be transparent as we go so that others can learn from both our process and the end product.
Thanks to Bill Penuel, Annie Allen, Jennifer Jacobs, Leah Teeter, Heather Hill, and Cynthia Pollard for their contributions to the observation protocol and feedback on this post.