INFORMS Open Forum

 Dear colleagues,  

We are writing this note to encourage constructive discussion of the refereeing process in Operations Research/ Management Science/ Operations Management.  The blog is based, first and foremost, upon our experience as authors, referees, editorial and advisory board members.  It is also based on discussions with many students and colleagues in conferences, seminars, and editorial board meetings.  We are also reflecting on recent efforts by editors of journals in the field to create a more coherent and constructive review process. 

Peer review is an important foundation on which our profession and its standard depend.  Teaching how to write a well-structured, fair, and constructive referee report is a crucial part of training Ph.D. students and post-docs.  Unfortunately, in our experience, young and even mature researchers often do not prepare such reports, which may result in the editorial process being more stressful than necessary for both authors and editors.  We feel that discussing the refereeing process is especially important as our community grows, in both membership and tools used, and as more and more authors and referees with different foundational education join it.  This growth results in a less unified refereeing process that we hope this note and the ensuing discussion will help to improve.  To be concrete, our note is written along the lines of the typical structure of a good referee report for a first-round review.  

1. Summary. An informative report should start with a summary paragraph.  This paragraph should summarize the paper and provide a critical appraisal of the problem under study, the modeling framework, and the main results.  The goal is to let the Associate Editor know the highlights of the paper as well as your overall assessment.  Too often we see summaries that imply the paper made lots of solid contributions only to find a negative recommendation such as reject or reject and resubmit.  For us the key components of an informative summary should be:  
1. Problem under study.  What is it? Is it important?  Is it within the scope of the journal?  
2. Modeling framework.  Describe the key parts of the framework (e.g., one supplier, one retailer, supplier sets the wholesale price, retailer responds with an order quantity and then sets the retail price, etc.).  Are there major issues with this framework i.e., are some parts inconsistent/ problematic?  For example, assuming the retailer is myopic while the supplier is strategic may not be a reasonable setup.  A good summary highlights aspects that are standard and those that are unique.  Obviously, the latter are more crucial and subject to more critical evaluation.  
3. Key results.  What are they claimed to be?  How significant are they (assuming they are correct)?  Evaluate the significance of the claimed contributions in view of the theme and quality of the journal.  Are there any issues with the correctness of the results?  If so, indicate which specific comments in part 3 below address these issues. 
2. Recommendation.  The most common recommendations in the first round are:   
1. Accept "As Is":  Neither one of us have seen this outcome on the first round, but legends are that it exists.  
2. Minor Revision:  You raise only a few easily addressable issues.  The crucial distinction from (c and d below) is that you see no need to review the paper again– it should be easy for the Associate Editor to ensure that the minor issues you raised have been addressed. 
3. Major Revision:  You see a clear path for publication in the journal, meaning that the issues you identified are fixable (at least in your mind).  You must outline a clear path to publication – i.e., what it would take (in terms of additional results, computational runs, justifications, etc.) to address the issues you raise and why you think this path is doable.  
4. Reject and Resubmit:  The meaning is similar to the major revision (c), except that you are less sure that the issues you raise are addressable (i.e., there is more risk involved in preparing an acceptable revision).  You must still indicate what it would take to have an acceptable version.  
5. Reject: If the issues you raise are very fundamental – e.g., you do not believe the basic model assumptions and/or the objective function are reasonable, there are technical errors that are not easily fixable, or the paper does not fit with the journal.  Then, this recommendation is required as there is no way to respond to these issues within the framework of the proposed model.  
3. Major comments. This part of the report should elaborate on your main criticisms of the paper and justify these criticisms.  Here are some do's and don'ts.  
1. Philosophical Objections.  Some reviews raise philosophical objections rather than ones specific to the paper.  An example is: "This paper adopts a monopolistic competition model. Such models are of limited interest."  An example of a more specific objection is: "This paper assumes that customers' valuation of service includes only travel cost, but not the cost of waiting."  A referee should not, as a rule, object to widely accepted frameworks or modeling approaches; such objections are more a matter of taste.   Just because a given framework is not your favorite, does not make it illegitimate, and should not be used as a basis for rejection.  Moreover, philosophical objectives are nearly impossible to address.  If they are to be used as a basis for rejection, this should be left up to the editorial team.  The referee's primary job is to point out potential issues specific to the paper.  Examples include:  
1. The paper assumed an infinite horizon objective for the supplier and established an asymptotic optimality result.  Both are extremely common frameworks in our community.  While one can question whether these apply in some practical situations, rejecting a paper because it adopts such standard frameworks is inappropriate, unless this framework is inadequate for the application under study (in which case the authors should be asked to provide a better justification for using this framework).  
2. The paper employs the framework of cooperative games.  Both cooperative and non-cooperative games are legitimate approaches.  In some settings, you can ask the authors to extend their results to the non-cooperative solution e.g., if it is an important benchmark, but rejecting the paper because you prefer the non-cooperative framework is inappropriate.  
3. The paper assumes infinite horizon objective for the supplier but a finite horizon objective for the retailer.  This is an inconsistency within the adopted framework and is a perfectly legitimate potential criticism of the paper – unless the authors can provide a strong justification for such dichotomous treatment of different players, this can be a basis for a major issue/ potential rejection. 
2. Assess the significance and validity of the claimed contributions.  Here the referee should, as much as possible, refrain from subjective judgement and point out how exactly the paper contributes to the current state of the art in this area.  Judgement of whether this contribution is of sufficient significance can be left to the editorial team.   
1. Are the key claimed contributions indeed established?  How do these contribute to the current state of the field?  Are there some extensions that must be made prior to publication?  Is yes, what are they?  Are they doable? Some examples: 
2. The paper considers a linear cost structure consisting of fixed plus variable costs.  Previous publications only considered variable costs.  Addition of the fixed cost component is significant because [describe the technical difficulties and new insights that this leads to]. 
3. The paper assumes customer demand is linear.  However, most of the results should also hold for convex demand functions.  I believe this extension should not lead to major technical difficulties and will significantly generalize the current framework.    
3. Outline concrete steps to bring the paper to a publishable level.  These should generally be divided into "must haves" (i.e., "minimal improvements" required to reach the publishable level) and "nice to have" ("desirable extensions" that would strengthen the paper). Specific questions are: 
1. How doable are the "must have" extensions? This will largely determine your recommendation. 
2. How doable are "nice to haves?" 
3. Are computational results sufficient?  If not, what should be added?  
4. Assess the technical correctness of the results.  This is, arguably, the main job of the referee, and is often the most time-consuming part of the review process.  The editors can decide on the legitimacy of basic assumptions and significance of contributions.  However, they do not have the time to go over the technical details and proofs.  Referees should clearly state that all technical results appear to be correct (if not, point out the issues found).  
5. Some referees recommend a rejection on the basis that the paper's key contributions are insufficient, and do not verify the correctness of the analysis (e.g., does the study addresses potential endogeneity).  Except in the most egregious cases, we recommend against this approach.  Review of the correctness of the results is important since: 
1. It often provides the authors with specific and constructive feedback. 

2. Your recommendations to the editor is easier to dismiss if it did not assess the technical correctness. 
3. Not assessing technical correctness may give us, as referees, the wrong incentives to try and reject papers based on more subjective grounds just to save time.  
3. Minor comments. List typos and grammatical mistakes as well as other relevant literature that may have been missed by the authors. The expected authors' response to most minor comments is "Done." 

To summarize:  

• We should try and review papers as if we wrote them, this way the feedback we provide to each other will help improve our papers and professional communities.  
• The referee should strive to provide objective feedback such as technical correctness of the results, placement of the results within the existing literature, etc. More subjective assessment (sufficiency of the level of contribution, fit to the journal, etc.) should generally be left to the editors. 

Opher Baron and Dmitry Krass, 

The Rotman School of Management, 

The University of Toronto.