by Holly Yettick
Reading research can sometimes feel like going to the gym. It’s probably a good idea, but it’s tempting to seek reasons why it’s not, especially when past experiences have proven to be time-consuming, boring, and less-than-productive.
Then there’s the question of where to start.
What research should you read? What should you do about highly-publicized concerns about social science research quality? It’s easy to throw up your hands and do the intellectual equivalent of skipping the crunches and heading for the ice cream parlor.
As an education researcher who believes that research can and should play some role in policy and practice, I feel frustrated when this occurs.
So, I recently wrote a very brief list of five tips to help policymakers more efficiently and effectively read and evaluate social science studies.
The list addresses five key issues to reading research: Peer review; The importance of prioritizing research reviews over standalone studies; “P values” and statistical significance; Effect sizes; and Research in real-world situations.
Here, I offer one place to start: a peer-refereed literature review.
Why literature reviews?
Regardless of size or scope, any one study can be anomalous or flawed. Literature reviews address this issue by summarizing the results of an entire body of research. When literature reviews use numbers to quantify the overall findings of a set of studies, they are called meta-analyses. Just as valuable are qualitative reviews that take stock of a body of research without quantifying it numerically. Such reviews can sometimes give you a better sense of nuance because they do avoid simplifying complex finding by distilling them into one or two numbers.
Why peer reviewed?
Simply put, peer review is a quality control measure.
“Peer review” is one of those terms that people may have heard without fully understanding. For example, one of the more common Google searches for Education Week is: “Is Education Week a peer-reviewed journal?” (It’s not; it’s a news outlet.) I think this may partially result from confusion about the definition of peer review.
So, I’ll explain how it works.
Scholars with expertise in the relevant field vet one another’s work, usually without being told who conducted the research to reduce the potential for personal bias. A peer-refereed journal is a publication that accepts or rejects work based on these blinded reviews by other experts. Often, articles that get accepted have been extensively revised and improved based on peer critiques. Books from scholarly presses can also be peer-reviewed, as can papers presented at academic conferences, grant proposals, and other bodies of work.
It’s easy to point to examples of shoddy peer-reviewed work and of rigorous studies that did not undergo peer review. And scholars themselves have advocated for changing and improving peer review.
The reality is that peer review is a little bit like a seat belt. It won’t save you every time but it will reduce the odds of reading low-quality research.
While less than perfect, the alternative (no quality control whatsoever) is probably worse.
Peer-reviewed journals vary in quality. There are multiple metrics to assess that quality including Thomson-Reuters impact factors and Elsevier’s Source Normalized Impact per Paper (SNIP). But probably the easiest way is to ask someone in the field, especially in education, where studies can appear in a wide variety of journals focusing on topics ranging from medicine to history to art.
For peer-reviewed literature reviews on education, my go-to source is the Review of Educational Research. Another oft-recommended source of reviews is the federal What Works Clearinghouse, which provides free access to research reviews that aim to be succinct and reader-friendly.
There’s been some attention in recent years to so-called “predatory journals” that pass themselves off as peer-refereed even though they’re not. Especially if you’re new to Research-land, stick to recommended journals with strong impact factors, and you should be safe.
I hit an expensive pay wall. Now what?
Almost every peer-reviewed publication lets you search for free, pulling up “abstracts”—article summaries that are about one paragraph long. But in order to read the entire article, you typically need to pay a considerable amount. If you end up using a lot of research (and I hope you will), you might want to consider subscribing to a journal or a database that catalogues multiple journals. (Think of the cost as a down payment on avoiding poor policy decisions!)
If not, try contacting the study authors directly. This can have the added benefit of giving you the opportunity to ask questions, or request a CliffsNotes version of the analysis. Researchers also sometimes post public versions of their articles on their personal, organizational or faculty webpages, or on Academia.edu.
In addition, you can follow researchers, research organizations, academic journals, and universities on social media, or sign up for email alerts.
By prioritizing peer-refereed research reviews rather than Googling for one-off studies, you will increase the odds of focusing on higher-quality studies while maximizing time savings by absorbing years-worth of findings in a single article.
Need more advice on interpreting what you find and applying it to the real world? You can find the original one-pager on the website of the National Education Policy Center at the University of Colorado Boulder School of Education, which commissioned the brief.