The Paper Reviewing ProcessPosted: October 18, 2013| Author:Nick Feamster|Filed under:advice, research, reviewing|
Learning how to review papers not only (obviously) makes you a better reviewer, but it can also help you as an author, since an understanding of the process can help you write your paper submissions for an audience of reviewers. If you know the criteria that a reviewer will use to judge your paper, you are in a much better position to tailor your paper so that it has a higher chance of being accepted.
There are many good resources that describe the paper reviewing process already, including those that explain the process (and its imperfections) and those that provide instructions for writing a good review (as well as techniques to avoid). There are also a few nice summaries of the review process for conferences in different areas of computer science that lend visibility into the process (e.g., here and here). Program committee chairs sometimes provide guidelines for writing reviews, such as these. I will not reiterate or summarize thosepreviousarticles here, but they are all definitely worth a read. Instead, I will discuss the importance of the review process and how it differs from simply reading a paper; I’ll also talk about how to prepare (and ultimately write) a review.
I will not talk about the paper selection process (i.e., what determines whether a paper is ultimately accepted or rejected), but will instead focus on the creation of a paper review. Program committee meetings are an important part of the paper selection process—at least in computer science—and I will be devoting a complete post to this topic next week. Meanwhile, I recommend reading Matt Welsh’s post on the psychology of program committees.
The Review Process
Why understanding the review process is important. Whether you end up reviewing a lot of papers as a Ph.D. student, your research will definitely be subject to the paper review process. It is imperative as a researcher to understand this process. Knowing the process can help you better write your paper for an audience of reviewers (and a program committee), and it can also help you maintain perspective when your paper is accepted or rejected. The process is far from perfect, and the outcome of the process is neither validation nor condemnation of your work. How you react—and how you adapt your research or follow through on it after the acceptance (or rejection)—is far more important to long-term success.
In the “Introduction to the Ph.D.” class at Georgia Tech, I ask students to create a research idea and write it up; a subsequent set of assignments asks the students to review and evaluate the ideas as part of a “mock” program committee. The process isn’t exactly the same as the review process for a full paper, but it is a lightweight way to have students experience the process first-hand in a low-stakes setting, and see both sides of the process (submission and review) at the same time. In next week’s blog post, I will discuss program committee meetings in general, as well as some observations from this year’s (and previous years’) in-class experiences with the mock PC.
Reviewing vs. reading. There are some significant distinctions between reading papers vs. reviewing them. When reading a paper for your own enrichment, your goal is to gather information as quickly as possible. In this case, you are a scientist who seeks to understand the context and content of existing work, to (for example) better understand how your own research might fit into the bigger picture or learn about techniques that might apply to your own work. The goal of reviewing is different. A reviewer’s goal is to first and foremost determine the suitability of a paper for some conference and second, to provide feedback to the authors to help them improve the paper in subsequent revisions. Remember that the reviewer’s primary goal trumps all other objectives: A reviewer often has a large number of papers to process and is typically not deeply devoted to improving the content of any particular paper. If you are lucky, you will get a diligent, thoughtful reviewer who provides thorough feedback, but do not be surprised if a review is not as thorough as you would have liked, or if the review “misses” some point you were trying to make. We would all like reviewers to make three passes through your paper submission—and, these are the instructions I would give, too, in an ideal world. Unfortunately, however, you will be lucky in many cases to get two thorough reads. The reviewer’s main goal is to determine the paper’s suitability for publication. As an author, you shouldn’t be surprised if some of the comments seem trivial: there may be underlying issues of taste that drove the reviewer’s opinion on your paper that a reviewer may not explicitly state. Whenever I read reviews I receive for a rejected paper, I try to look past specific detailed quibbles (or “excuses” for rejecting the paper) and figure out the big picture: the reviewer couldn’t find a reason to accept the paper.
Calibration: Reviewing one paper vs. reviewing many papers. The paper review process can differ depending on who, exactly, is reviewing the paper. For example, as a Ph.D. student, you may review one or two papers at a time, as an “external reviewer” for a conference or journal. Journal editors and program committee chairs often seek the help of external reviewers if they need a particular subject-matter expert to review a paper. Later in your Ph.D. career, you may have established yourself as an expert on a particular topic and find yourself reviewing a paper here and there on a handful of topics. Sometimes a member of the program committee (e.g., your advisor) might ask you to help review a particular paper. As you progress in your career, you will be asked to serve on program committees yourself, whereupon you’ll find yourself with tens of papers to review over the course of a couple of months. Ironically, it is sometimes easier to review a group of papers than a single (or a few) papers, because seeing a group of papers helps you “calibrate” your scores and rankings of papers according to the general quality of papers that have been submitted to the conference. If you have been asked to review a single paper for a conference, you should either figure out how to calibrate your assessment with respect to other papers that might have been submitted, or simply review the paper on its merits while reserving judgement as to the paper’s ultimate disposition.
Does the Paper Realize a Great Idea?
Look for a reason to accept the paper. Does it realize a great contribution or idea? Every paper is imperfect. The paper may have made an incorrect or imperfect assumption. The experiments may not have been as thorough as you liked. The graphs may be difficult to read. Parts of the paper may be difficult to understand. These types of issues certainly reflect problems with a paper, but they do not necessarily constitute a reason to reject a paper if they do not affect the correctness or significance of the main underlying conclusion or contribution of the paper. Therefore, the first two questions I ask myself when reviewing a paper are: (1) Does the paper have a great idea?; and (2) Does it realize the great idea? (or, alternatively, to what extent does it realize that great idea, since typically no paper is water-tight).
What makes an idea “great”?Judging a paper’s contribution turns out to be highly subjective, which is why the review process remains so uncertain. A paper isn’t judged on a set of fixed checkboxes, a grading “key”, or any notion of absolute correctness. Reviewers often reserve considerable judgment based on “taste“, and reasonable people will disagree as to the merits of the main contribution or idea in a paper. In fact, there has been a fair amount of documentation that, as reviewers, we are often quite terrible at predicting the merits of a particular piece of submitted work: There’s a great article on this topic, as well as some parodies to illustrate the subjective nature of the process. Many fields have also introduced a “test of time” award to papers from past decades, to recognize accepted papers that have truly had long-term positive impact (implicitly acknowledging that this is almost impossible to assess when a paper is first published). Due to the subjective nature of this judgment, it is all the more important that your writing is clear, and well-matched to what a reviewer is looking for (i.e., the contributions and ideas).
Invariant questions. Different conferences may have different value structures, and the chairs of any given conference may ask the reviewers to focus on different criteria when judging a paper. Regardless, there are some invariant questions that most reviewers would (or at least should) always consider, including:
- Is the problem important? What problem is the paper trying to solve, and is it important? Seek to summarize the paper’s contribution in one sentence. Make this short summary the beginning of your review, as well. Try to convince yourself (by reading the paper or otherwise) that a solution to the problem that the paper is proposing would advance knowledge or significantly improve the state of affairs for some group of people. Note that you may not care about the problem, but also ask yourself whether you can imagine some group of readers who will be interested in the solution to the problem. When asking yourself this question about a paper, try to divorce your own taste about the problem’s importance from the more general question concerning whether there is some group of people who would be interested in the problem the paper is addressing and solving.
- To what extent does the paper solve the problem it describes? A single paper very rarely closes the book on a single problem, but it may take an important step towards solving the problem. It might solve the problem for an important set of operating conditions or under a new set of assumptions. Or, if the problem area is completely new, perhaps the paper doesn’t really solve the problem at all, but simply articulating a new problem area for follow-on work is a significant contribution.
- What is the “intellectual nugget”? As a reviewer, I try to identify whether a paper has a particular intellectual kernel that lies at the heart of the solution. This kernel is often what separates an important research contribution from a simple matter of engineering. This intellectual nugget might be the application (or invention) of a particular technique, a proof of correctness (where one previously did not exist), or an attempt to put the solution into a broader intellectual context. In other words, the intellectual contribution might be to take a general problem and tackle a specific sub-problem (e.g., under certain assumptions or conditions), or to take a specific problem and generalize it (e.g., develop a general theory, proof of correctness, or taxonomy). Looking through the paper for applications of specific research patterns can help identify an intellectual nugget, if one exists.
- What is the main contribution or conclusion? Is it important? As a reviewer, I try to concisely articulate the paper’s main contribution (or small number of contributions). Often, a paper will helpfully summarize those contributions somewhere in the introduction (Jim Kurose’s advice on writing paper introductions advises the writer to explicitly do so). The reviewer’s job is then to assess whether those contributions are significant or important enough to warrant a publication. The significance of those contributions often depends on the perceived increment over previous work. All work is incremental to some degree, as everything builds on past work. The author’s job is to convince the reviewer that the increment is important, and the reviewer’s job is to assess the author’s claims of significance.
- Does the content support the conclusion? An introduction may make broad (or wild) claims, and it is important to dig into the paper to determine whether the content of the paper supports the conclusion. Are the experiments run correctly? Are they based on the correct set of assumptions? If the conclusion involves comparison to previous work, is the comparison performed in a controlled manner, using an equivalent (or at least fair) experimental setup? If applicable, have the authors released their code and data so that you (or others) can check the claims yourself?
Preparing Your Review
Consider the audience. Not every publication venue is the same. Some venues are explicitly geared towards acceptance of early, incomplete work that is likely to generate discussion (many workshops use this criterion for acceptance). Other venues favor contributions that constitute well-executed, smaller increments. When reviewing a paper, either externally or as a member of a committee, your first question should be to consider the audience for the conference, workshop, or journal, and whether the likely audience for the venue would benefit from reading the paper. The question of audience involves that of both the “bar” for acceptance (Does the paper meet the audience’s standards for something that is worth reading?) and the “scope” of the venue (is the paper on-topic for the venue?). Often, scope can be (and is) broadly construed, so the key question really boils down to whether the likely audience for the paper will benefit from reading it.
Consider the standards. Your standards will (and should) vary depending on the venue for which you are reviewing a paper submission. Workshops are typically more permissive as far as accepting “vision” papers that outline a new problem or problem area or papers that “foster discussion” than conferences, which typically aim to accept more complete pieces of work. Nevertheless, even the standards for a conference review process will vary depending on both the conference itself, the program committee chair’s instructions about how permissive to be, and the relative quality of the group of papers that you are reviewing. A good way to get a sense for the standards of a conference for which you are reviewing is to read through the complete set of papers that you have been asked to review and rank them, before writing a single review. This will ensure some level of calibration, although it is still biased based on the set of papers that you are reviewing. Reading past proceedings of the particular journal or conference can also help you determine the appropriate standard to set for acceptance.
Consider the purpose. Different papers serve different purposes. Multiple paper submissions to the same venue might in fact have quite different purposes, and it is important to establish what the paper is contributing (or attempting to contribute) before passing judgement. For example, a paper might be a complete piece of work, but it might also be a survey, a tutorial, or simply a proposal. If the paper is one of the latter types, your first questions as a reviewer should concern whether the audience would benefit from the survey, tutorial, or proposal, and whether such a paper meets the standards for the conference. If the answers to those questions are “yes”, then your evaluation should be tailored to the paper’s purpose. If the paper is a survey, your assessment should be based on the completeness of the survey, with respect to the area that the paper is claiming to summarize. If the paper is a tutorial, is the description correct and clearly described? If the paper is a proposal, does the proposed research agenda make sense, and is the outcome (if the proposal is successful) worthwhile?
Consider the big picture. Every paper can be rejected. It is always easy to find reasons to reject a paper. The reviewer’s goal should not be to identify the reasons to reject a paper, but rather to determine whether there are any reasons to accept the paper. If the answer to that question is negative, then it is always easy to find “excuses” to reject a paper (recall the discussion above). You should be aiming to figure out whether the paper has important contributions that the audience will benefit from knowing about, and whether the paper supports those contributions and conclusions to the level of standard that is commensurate with the standard of the audience and the venue. One litmus test I use to ensure that a negative aspect of a paper does not condemn it is to ask myself whether the problem (1) affects the main conclusion or contribution of the paper; and (2) can be fixed easily in a revision. If the problem doesn’t affect the main contribution or conclusion, and if it can be easily fixed, then it should not negatively affect a paper’s review.
Writing Your Review
Start with a summary of the paper and its contributions. A short, one-paragraph summary describing the paper’s main contribution(s) demonstrates to the authors (and to you!) that you understand the main point of the paper. This helps you as a reviewer articulate the main contributions and conclusions of the paper for the purposes of your own evaluation. Try to address the type of paper it is (is it a survey paper, for example?), the context for the paper (i.e., how it builds on or relates to previous work), its overall correctness, and its contributions. If you cannot concisely summarize the paper, then the paper is not in good shape, and you can reflect this assessment in the review, as well. These summaries are very helpful to authors, since they may not match the authors’ views of the main contribution! For example, as an author, you can easily figure out if you’ve “missed the mark” or whether the reviewer fundamentally misunderstood the paper by reading a reviewer’s summary of your own work. If the summary of the contribution does not match your own view of the paper’s contribution, then you know that you have some work to do in writing and presentation.
Assess whether the paper delivers on the main claims and contributions. You should provide an assessment, for each of the paper’s main claims and contributions, whether it delivers on that claim. If the main contribution of the paper is flawed, you should indicate whether you think a flaw is “fatal”, or whether the authors could simply fix the flaw in a revision if the paper is accepted. Sometimes flaws (e.g., inconsistent terminology) are fixable. Other flaws (e.g., a questionable experimental setup) may or may not be fixable. While it might seem that a broken experimental setup is “fatal”, ask yourself as a reviewer whether the conclusions from the paper’s experiments as is are still meaningful, even if the authors have not interpreted the results correctly. If the conclusions from the experiments can be restated and still turn out to be meaningful contributions—or, if the flaw in an experiment doesn’t affect the main contribution or conclusion—then even a flaw in experiments can likely be fixed in revision. Occasionally, however, experiments may need to be completely redesigned because they don’t support any meaningful conclusion. Or, the content of the paper may simply be incorrect; sometimes correctness issues are difficult for a reviewer to spot, so a paper isn’t necessarily “correct” simply because a reviewer has validated the paper. Regardless, if there are correctness issues that affect the main contribution of the paper that call into question whether the main result or contribution is correct in the first place, the paper’s review should reflect these concerns and likely cannot be accepted.
Discuss positive aspects of the paper; always try to find something positive, even in “bad” papers. It is easy to identify problems with a paper. It can be much trickier (especially with “average” papers) to identify the positive aspects and contributions, but most papers typically have at least some small kernel of goodness. Even for particularly bad papers, there might be one sentence in the introduction, discussion, or future work section that makes an interesting point or highlights a possibility for interesting contributions. In a pinch, if you can’t find anything positive, those are good places to look. As a reviewer, you can remark that those observations are interesting, and that you would really like to see those parts of the work further developed. These positive comments aren’t just for author morale (although that’s important, too): They give the author a direction to move forward. The worst reviews are those that reject a paper but don’t provide any specific action for moving forward. The best reviews are those that highlight the positive aspects of the work, while identifying weaknesses and areas where the work could be further developed to address weaknesses or build on the paper’s existing strengths.
Criticize the paper, not the authors. When writing your review, consider the type of review that you would like to receive. Always be polite, respectful, and positive. Don’t be personal. Choose your language carefully, as it will help convey your message. For example, if you say “the authors don’t consider the related work”, that is a much more personal statement than “the paper doesn’t consider the related work”. (In fact, you don’t know if the authors considered a particular piece of related work anyway; they may have simply chosen not to include it in the writeup!) Talking about “the authors” gets personal, and it will put the authors themselves on the defensive when reading your review. Instead, focus on “the paper” and frame your critique around “suggestions for improvement”. Never, ever insult the authors; don’t accuse the authors of being sloppy or unethical researchers. As a reviewer, you don’t always know the full context, so limit your judgement to what you can directly conclude by reading the paper.
Consider the type of feedback you would like to receive. Receiving reviews for rejected papers is a part of the research process, but it is never fun for the authors (particularly new Ph.D. students). Do your part to contribute positively to the process by suggesting changes that you’d like to see if you had to review the paper again. In all likelihood, you may see the paper again in the form of a revision!
"How to" Guideline series is coordinated by Helen Mongan-Rallis of the Education Department at the University of Minnesota Duluth. If you have any questions, comments, or suggestions to improve these guidelines please me at e-mail firstname.lastname@example.org.
Guidelines for writing a literature review
by Helen Mongan-Rallis. Last updated: November 21, 2014
[Note: For these guidelines, in some sections I have quoted directly some of the the steps from: Galvan, J. (2006). Writing literature reviews: a guide for students of the behavioral sciences (3rd ed.). Glendale, CA: Pyrczak Publishing.]
What is a literature review?
A literature review is not an annotated bibliography in which you summarize briefly each article that you have reviewed. While a summary of the what you have read is contained within the literature review, it goes well beyond merely summarizing professional literature. It focuses on a specific topic of interest to you and includes a critical analysis of the relationship among different works, and relating this research to your work. It may be written as a stand-alone paper or to provide a theoretical framework and rationale for a research study (such as a thesis or dissertation).
These guidelines are adapted primarily from Galvan (2006). Galvan outlines a very clear, step-by-step approach that is very useful to use as you write your review. I have integrated some other tips within this guide, particularly in suggesting different technology tools that you might want to consider in helping you organize your review. In the sections from Step 6-9 what I have included is the outline of those steps exactly as described by Galvan. I also provide links at the end of this guide to resources that you should use in order to search the literature and as you write your review.
In addition to using the step-by-step guide that I have provided below, I also recommend that you (a) locate examples of literature reviews in your field of study and skim over these to get a feel for what a literature review is and how these are written (I have also provided links to a couple of examples at the end of these guidelines (b) read over other guides to writing literature reviews so that you see different perspectives and approaches: Some examples are:
- Review of Literature: University of Wisconsin - Madison The Writing Center.
- How to ..Write a Literature Review: University of California, Santa Cruz University Library).
- Information Fluency - Literature Review: Washington & Lee University
- How to Do A Literature Review? North Carolina A&T State University F.D. Bluford Library.
- Selected Links to Resources on Writing a Literature Review
Step 1: Review APA guidelines
Read through the links provided below on APA guidelines so that you become familiar with the common core elements of how to write in APA style: in particular, pay attention to general document guidelines (e.g. font, margins, spacing), title page, abstract, body, text citations, quotations.
Step 2: Decide on a topic
It will help you considerably if your topic for your literature review is the one on which you intend to do your final M.Ed. project, or is in some way related to the topic of your final project. However, you may pick any scholarly topic.
Step 3: Identify the literature that you will review:
- Familiarize yourself with online databases (see UMD library resource links below for help with this), identifying relevant databases in your field of study.
- Using relevant databases, search for literature sources using Google Scholar and also searching using Furl (search all sources, including the Furl accounts of other Furl members). Some tips for identifying suitable literature and narrowing your search :
- Start with a general descriptor from the database thesaurus or one that you know is already a well defined descriptor based on past work that you have done in this field. You will need to experiment with different searches, such as limiting your search to descriptors that appear only in the document titles, or in both the document title and in the abstract.
- Redefine your topic if needed: as you search you will quickly find out if the topic that you are reviewing is too broad. Try to narrow it to a specific area of interest within the broad area that you have chosen (remember: this is merely an introductory literature review for Educ 7001). It is a good idea, as part of your literature search, to look for existing literature reviews that have already been written on this topic.
- As part of your search, be sure to identify landmark or classic studies and theorists as these provide you with a framework/context for your study.
- Import your references into your RefWorks account (see: Refworks Import Directions for guide on how to do this from different databases). You can also enter references manually into RefWorks if you need to.
Step 4: Analyze the literature
Once you have identified and located the articles for your review, you need to analyze them and organize them before you begin writing:
- Overview the articles: Skim the articles to get an idea of the general purpose and content of the article (focus your reading here on the abstract, introduction and first few paragraphs, the conclusion of each article. Tip: as you skim the articles, you may want to record the notes that you take on each directly into RefWorks in the box for User 1. You can take notes onto note cards or into a word processing document instead or as well as using RefWorks, but having your notes in RefWorks makes it easy to organize your notes later.
- Group the articles into categories (e.g. into topics and subtopics and chronologically within each subtopic). Once again, it's useful to enter this information into your RefWorks record. You can record the topics in the same box as before (User 1) or use User 2 box for the topic(s) under which you have chosen to place this article.
- Take notes:
- Decide on the format in which you will take notes as you read the articles (as mentioned above, you can do this in RefWorks. You can also do this using a Word Processor, or a concept mapping program like Inspiration (free 30 trial download), a data base program (e.g. Access or File Maker Pro), in an Excel spreadsheet, or the "old-fashioned" way of using note cards. Be consistent in how you record notes.
- Define key terms: look for differences in the way keys terms are defined (note these differences).
- Note key statistics that you may want to use in the introduction to your review.
- Select useful quotes that you may want to include in your review. Important: If you copy the exact words from an article, be sure to cite the page number as you will need this should you decide to use the quote when you write your review (as direct quotes must always be accompanied by page references). To ensure that you have quoted accurately (and to save time in note taking), if you are accessing the article in a format that allows this, you can copy and paste using your computer "edit --> copy --> paste" functions. Note: although you may collect a large number of quotes during the note taking phase of your review, when you write the review, use quotes very sparingly. The rule I follow is to quote only when when some key meaning would be lost in translation if I were to paraphrase the original author's words, or if using the original words adds special emphasis to a point that I am making.
- Note emphases, strengths & weaknesses: Since different research studies focus on different aspects of the issue being studied, each article that you read will have different emphases, strengths. and weaknesses. Your role as a reviewer is to evaluate what you read, so that your review is not a mere description of different articles, but rather a critical analysis that makes sense of the collection of articles that you are reviewing. Critique the research methodologies used in the studies, and distinguish between assertions (the author's opinion) and actual research findings (derived from empirical evidence).
- Identify major trends or patterns: As you read a range of articles on your topic, you should make note of trends and patterns over time as reported in the literature. This step requires you to synthesize and make sense of what you read, since these patterns and trends may not be spelled out in the literature, but rather become apparent to you as you review the big picture that has emerged over time. Your analysis can make generalizations across a majority of studies, but should also note inconsistencies across studies and over time.
- Identify gaps in the literature, and reflect on why these might exist (based on the understandings that you have gained by reading literature in this field of study). These gaps will be important for you to address as you plan and write your review.
- Identify relationships among studies: note relationships among studies, such as which studies were landmark ones that led to subsequent studies in the same area. You may also note that studies fall into different categories (categories that you see emerging or ones that are already discussed in the literature). When you write your review, you should address these relationships and different categories and discuss relevant studies using this as a framework.
- Keep your review focused on your topic: make sure that the articles you find are relevant and directly related to your topic. As you take notes, record which specific aspects of the article you are reading are relevant to your topic (as you read you will come up with key descriptors that you can record in your notes that will help you organize your findings when you come to write up your review). If you are using an electronic form of note taking, you might note these descriptors in a separate field (e.g. in RefWorks, put these under User 2 or User 3; in Excel have a separate column for each descriptor; if you use Inspiration, you might attach a separate note for key descriptors.
- Evaluate your references for currency and coverage: Although you can always find more articles on your topic, you have to decide at what point you are finished with collecting new resources so that you can focus on writing up your findings. However, before you begin writing, you must evaluate your reference list to ensure that it is up to date and has reported the most current work. Typically a review will cover the last five years, but should also refer to any landmark studies prior to this time if they have significance in shaping the direction of the field. If you include studies prior to the past five years that are not landmark studies, you should defend why you have chosen these rather than more current ones.
Step 5: Summarize the literature in table or concept map format
- Galvan (2006) recommends building tables as a key way to help you overview, organize, and summarize your findings, and suggests that including one or more of the tables that you create may be helpful in your literature review. If you do include tables as part of your review each must be accompanied by an analysis that summarizes, interprets and synthesizes the literature that you have charted in the table. You can plan your table or do the entire summary chart of your literature using a concept map (such as using Inspiration)
- You can create the table using the table feature within Microsoft Word, or can create it initially in Excel and then copy and paste/import the the Excel sheet into Word once you have completed the table in Excel. The advantage of using Excel is that it enables you to sort your findings according to a variety of factors (e.g. sort by date, and then by author; sort by methodology and then date)
- Examples of tables that may be relevant to your review:
- Definitions of key terms and concepts.
- Research methods
- Summary of research results
Step 6: Synthesize the literature prior to writing your review
Using the notes that you have taken and summary tables, develop an outline of your final review. The following are the key steps as outlined by Galvan (2006: 71-79)
- Consider your purpose and voice before beginning to write. In the case of this Educ 7001 introductory literature review, your initial purpose is to provide an overview of the topic that is of interest to you, demonstrating your understanding of key works and concepts within your chosen area of focus. You are also developing skills in reviewing and writing, to provide a foundation on which you will build in subsequent courses within your M.Ed. and ultimately in your final project. In your final project your literature review should demonstrate your command of your field of study and/or establishing context for a study that you have done.
- Consider how you reassemble your notes: plan how you will organize your findings into a unique analysis of the picture that you have captured in your notes. Important: A literature review is not series of annotations (like an annotated bibliography). Galvan (2006:72) captures the difference between an annotated bibliography and a literature review very well: "...in essence, like describing trees when you really should be describing a forest. In the case of a literature review, you are really creating a new forest, which you will build by using the trees you found in the literature you read."
- Create a topic outline that traces your argument: first explain to the reader your line or argument (or thesis); then your narrative that follows should explain and justify your line of argument. You may find the program Inspiration useful in mapping out your argument (and once you have created this in a concept map form, Inspiration enables you to convert this to a text outline merely by clicking on the "outline" button). This can then be exported into a Microsoft Word document.
- Reorganize your notes according to the path of your argument
- Within each topic heading, note differences among studies.
- Within each topic heading, look for obvious gaps or areas needing more research.
- Plan to describe relevant theories.
- Plan to discuss how individual studies relate to and advance theory
- Plan to summarize periodically and, again near the end of the review
- Plan to present conclusions and implications
- Plan to suggest specific directions for future research near the end of the review
- Flesh out your outline with details from your analysis
Step 7: Writing the review (Galvan, 2006: 81-90)
- Identify the broad problem area, but avoid global statements
- Early in the review, indicate why the topic being reviewed is important
- Distinguish between research finding and other sources of information
- Indicate why certain studies are important
- If you are commenting on the timeliness of a topic, be specific in describing the time frame
- If citing a classic or landmark study, identify it as such
- If a landmark study was replicated, mention that and indicate the results of the replication
- Discuss other literature reviews on your topic
- Refer the reader to other reviews on issues that you will not be discussing in details
- Justify comments such as, "no studies were found."
- Avoid long lists of nonspecific references
- If the results of previous studies are inconsistent or widely varying, cite them separately
- Cite all relevant references in the review section of thesis, dissertation, or journal article
Step 8: Developing a coherent essay (Galvan, 2006: 91-96)
- If your review is long, provide an overview near the beginning of the review
- Near the beginning of a review, state explicitly what will and will not be covered
- Specify your point of view early in the review: this serves as the thesis statement of the review.
- Aim for a clear and cohesive essay that integrates the key details of the literature and communicates your point of view (a literature is not a series of annotated articles).
- Use subheadings, especially in long reviews
- Use transitions to help trace your argument
- If your topic teaches across disciplines, consider reviewing studies from each discipline separately
- Write a conclusion for the end of the review: Provide closure so that the path of the argument ends with a conclusion of some kind. How you end the review, however, will depend on your reason for writing it. If the review was written to stand alone, as is the case of a term paper or a review article for publication, the conclusion needs to make clear how the material in the body of the review has supported the assertion or proposition presented in the introduction. On the other hand, a review in a thesis, dissertation, or journal article presenting original research usually leads to the research questions that will be addressed.
- Check the flow of your argument for coherence.
Galvan, J. (2006). Writing literature reviews: a guide for students of the behavioral sciences ( 3rd ed.). Glendale, CA: Pyrczak Publishing.
- UMD & library resources and links:
- UMD library research tools: includes links to
- Refworks Import Directions: Links to step-by-step directions on how to important to Refworks from different databases
- Writing guidelines:
- Purdue OWL (Online Writing Lab): A user-friendly writing lab that parallels with the 5th edition APA manual.
- APA guidelines:
- APA Style Essentials: overview of common core of elements of APA style.
- APA Research Style Crib Sheet is a summary of rules for using APA style.
- APA Style for Electronic Media and URL's: commonly asked questions regarding how to cite electronic media
- Examples of literature reviews:
- Johnson, B. & Reeves, B. (2005). Challenges. Literature review chapter from unpublished master's thesis, University of Minnesota Duluth, Minnesota.
- Maguire, L. (2005). Literature review – faculty participation in online distance education: barriers and motivators. Online Journal of Distance Learning Administration, Volume 8, No. 1, Spring 2005. State University of West Georgia, Distance Education Center.
Return to the Index of How To Guidelines