INFORMS Open Forum

The Psychology of a Reviewer in Operations Management

By Brad Staats and Christopher Tang

Over the last 20 years, two surprising trends in top OM research journals have emerged:

1)      The length of reviewers’ reports are getting longer; and

2)      The tone of the reviewers’ reports are getting more negative. 

However, it does not have to be this way. These trends are not prevalent in engineering (except Industrial Engineering), Marketing, Economics, etc.   Our community is comprised of friendly colleagues who are social and caring.  Ask someone for help on a paper at a conference and you’ll likely get thoughtful well-meaning advice. However, when it becomes a review, something changes. Why is this happening? 

As OM scholars, we know that before solving a problem we have to get to the root cause. Thus, if we want to develop proactive steps to energize our OM research community, we need to understand our limitations. Through our informal discussions with PhD students, junior and senior colleagues over the years, we have identified the following root causes that need our attention:

• Broadening scope.    As OM research expanded its scope, researchers began to explore diverse operational issues arising in different countries, industries, and settings by using varied research methods (analytical, empirical, experimental).  This is exciting news, but it is difficult for reviewers to know the importance of all research questions arising in each of the different contexts.  Consequently, many reports are long because, without a clear understanding of the contexts, the reviewers may raise questions or even doubts about the contexts or may have different views about research questions or even methodologies.  Thoughtful doubts are helpful to aid the author in manuscript development, however, when there are doubts, the negativity may come through unintentionally. 
• Reviewer’s self-perception.   For most OM journals, reviewers observe reports written by other reviewers.  Thus, long and negative reports can become self-fulfilling. Unless editors take the leadership to provide a more balanced view, some reviewers may think their own standards are “too low” or their own reports being “too positive.”  Consequently, more reviewers want to become more demanding by imposing higher standards so as to ensure that they are being seen as solid scholars.  This may explain why most reports are long and negative.  
• Author’s tit for tat.  When authors experience longer and negative reviews themselves, they may think this is the norm.  Consequently, they behave the same way when they review others’ papers.  
• Fierce competition.  Over the last 20 years, the number of business schools around the world (especially China, Europe and India) has grown but the number of top OM research journals remains the same. Other domains tend to have more top journals, for example, Finance and Marketing have at least 5 top journals in their respective fields.  Consequently, the competition among OM researchers is growing.  As more business schools demand publications in top OM journals, the perceived mission of the journals has changed from knowledge dissemination to “promotion criteria.”   Fierce competition can cause our colleagues to become more negative.
• Incentive bias.  Research shows that individuals worry about what others think of them. There is no reason to think that OM scholars should be any different. Thus, reviewers worry about the perception that editors have from what they write. Given that almost all papers have flaws there is little risk in rejecting a “good” paper. However, a reviewer may see a perceived penalty for accepting a “bad” paper.  Hence, it is safer to reject than to accept.  Also, to show that a reviewer is a thoughtful scholar it is safe for reviewers to demand many different extensions, empirical analysis, or experiments to ensure the results are robust – far exceeding what they would expect from their own work. We have talked with many colleagues who say something along the lines of, “you can’t just say that it is good, even if it is – you have to ask for something more.” This may explain what most reports are long and negative. 

These trends and the underlying root causes highlight two truths about the review process. First, we do want to screen out papers that lack significant contributions. Therefore, reviewers rigorously engaging with a paper does just that and so should be encouraged. However, as a second point – these same traits can hinder innovative ideas that can stimulate exciting research. It may appear safer to reject something that looks different, but unless we allow some papers containing innovative ideas to kick start new topics in OM, our research will become stagnant.  

What can we do as a community?   We welcome your suggestions.

As we search for ideas to build a vibrant OM research community, we learned that Computer Science faced the same challenge. Ed Catmull -- a computer scientist who co-founded Pixar[1] – wrote about these challenges in his book “Creativity, Inc.” Discussing his experience when serving on a committee for accepting papers for a computer graphics conference, he noted: 

“At each of the meetings, I was struck that there seemed to be two kinds of reviewers: some who would look for flaws in the papers, and then pounce to kill them; and others who started from a place of seeking and promoting good ideas. When the “idea protectors”saw flaws, they pointed them out gently, in the spirit of improving the paper—not eviscerating it. Interestingly, the “paper killers” were not aware that they were serving some other agenda (which was often, in my estimation, to show their colleagues how high their standards were). Both groups thought they were protecting the proceedings, but only one group understood that by looking for something new and surprising, they were offering the most valuable kind of protection. Negative feedback may be fun, but it is far less brave than endorsing something unproven and providing room for it to grow.”

Authors, reviewers, and editors need to have this courage for our field to thrive. At each step in the review process, individuals should be asking, as Catmull suggests, what is the good idea in this paper, rather than how will my answer be perceived by others? An honest and open dialogue with all of our ideas permits us to improve the efficiency of our review process and the effectiveness of our research creation. We believe that there is something we can learn from Ed Catmull: it takes courage to embrace something we are not familiar with, but in so doing, we all learn and improve.

[1] Ed Catmull was President and CTO of Pixar who developed the RenderMan rendering system used in films such as Toy Story and Finding Nemo.  He is now President of Walt Disney and Pixar Animation Studios.

------------------------------
Christopher Tang
Professor
University of California-Los Angeles
Los Angeles CA
------------------------------

Excellent article.

We are practitioners, not academics. We don't really need to publish peer reviewed work, but we do. We encourage our younger folks to do it, but our senior contributors are less likely to bother.

One reason our senior staff members do less peer reviewed work is the hassle of being criticized by a younger reviewer with only academic experience. Often our professionals have experienced several cycles of learning in multiple disciplines. The younger purists lack the wisdom to know what they don't know.

So, in particular we see multidisciplinary works as difficult.

It is comforting to remember that Kahneman and Teversky's Nobel winning work in Prospect Theory was rejected by the first journal review, and it appeared in another publication instead.

Steve

214-435-8334
www.lone-star.com


Sent from my iPad


Original Message------

The Psychology of a Reviewer in Operations Management

By Brad Staats and Christopher Tang

Over the last 20 years, two surprising trends in top OM research journals have emerged:

1)      The length of reviewers’ reports are getting longer; and

2)      The tone of the reviewers’ reports are getting more negative. 

However, it does not have to be this way. These trends are not prevalent in engineering (except Industrial Engineering), Marketing, Economics, etc.   Our community is comprised of friendly colleagues who are social and caring.  Ask someone for help on a paper at a conference and you’ll likely get thoughtful well-meaning advice. However, when it becomes a review, something changes. Why is this happening? 

As OM scholars, we know that before solving a problem we have to get to the root cause. Thus, if we want to develop proactive steps to energize our OM research community, we need to understand our limitations. Through our informal discussions with PhD students, junior and senior colleagues over the years, we have identified the following root causes that need our attention:

• Broadening scope.    As OM research expanded its scope, researchers began to explore diverse operational issues arising in different countries, industries, and settings by using varied research methods (analytical, empirical, experimental).  This is exciting news, but it is difficult for reviewers to know the importance of all research questions arising in each of the different contexts.  Consequently, many reports are long because, without a clear understanding of the contexts, the reviewers may raise questions or even doubts about the contexts or may have different views about research questions or even methodologies.  Thoughtful doubts are helpful to aid the author in manuscript development, however, when there are doubts, the negativity may come through unintentionally. 
• Reviewer’s self-perception.   For most OM journals, reviewers observe reports written by other reviewers.  Thus, long and negative reports can become self-fulfilling. Unless editors take the leadership to provide a more balanced view, some reviewers may think their own standards are “too low” or their own reports being “too positive.”  Consequently, more reviewers want to become more demanding by imposing higher standards so as to ensure that they are being seen as solid scholars.  This may explain why most reports are long and negative.  
• Author’s tit for tat.  When authors experience longer and negative reviews themselves, they may think this is the norm.  Consequently, they behave the same way when they review others’ papers.  
• Fierce competition.  Over the last 20 years, the number of business schools around the world (especially China, Europe and India) has grown but the number of top OM research journals remains the same. Other domains tend to have more top journals, for example, Finance and Marketing have at least 5 top journals in their respective fields.  Consequently, the competition among OM researchers is growing.  As more business schools demand publications in top OM journals, the perceived mission of the journals has changed from knowledge dissemination to “promotion criteria.”   Fierce competition can cause our colleagues to become more negative.
• Incentive bias.  Research shows that individuals worry about what others think of them. There is no reason to think that OM scholars should be any different. Thus, reviewers worry about the perception that editors have from what they write. Given that almost all papers have flaws there is little risk in rejecting a “good” paper. However, a reviewer may see a perceived penalty for accepting a “bad” paper.  Hence, it is safer to reject than to accept.  Also, to show that a reviewer is a thoughtful scholar it is safe for reviewers to demand many different extensions, empirical analysis, or experiments to ensure the results are robust – far exceeding what they would expect from their own work. We have talked with many colleagues who say something along the lines of, “you can’t just say that it is good, even if it is – you have to ask for something more.” This may explain what most reports are long and negative. 

These trends and the underlying root causes highlight two truths about the review process. First, we do want to screen out papers that lack significant contributions. Therefore, reviewers rigorously engaging with a paper does just that and so should be encouraged. However, as a second point – these same traits can hinder innovative ideas that can stimulate exciting research. It may appear safer to reject something that looks different, but unless we allow some papers containing innovative ideas to kick start new topics in OM, our research will become stagnant.  

What can we do as a community?   We welcome your suggestions.

As we search for ideas to build a vibrant OM research community, we learned that Computer Science faced the same challenge. Ed Catmull -- a computer scientist who co-founded Pixar[1] – wrote about these challenges in his book “Creativity, Inc.” Discussing his experience when serving on a committee for accepting papers for a computer graphics conference, he noted: 

“At each of the meetings, I was struck that there seemed to be two kinds of reviewers: some who would look for flaws in the papers, and then pounce to kill them; and others who started from a place of seeking and promoting good ideas. When the “idea protectors”saw flaws, they pointed them out gently, in the spirit of improving the paper—not eviscerating it. Interestingly, the “paper killers” were not aware that they were serving some other agenda (which was often, in my estimation, to show their colleagues how high their standards were). Both groups thought they were protecting the proceedings, but only one group understood that by looking for something new and surprising, they were offering the most valuable kind of protection. Negative feedback may be fun, but it is far less brave than endorsing something unproven and providing room for it to grow.”

Authors, reviewers, and editors need to have this courage for our field to thrive. At each step in the review process, individuals should be asking, as Catmull suggests, what is the good idea in this paper, rather than how will my answer be perceived by others? An honest and open dialogue with all of our ideas permits us to improve the efficiency of our review process and the effectiveness of our research creation. We believe that there is something we can learn from Ed Catmull: it takes courage to embrace something we are not familiar with, but in so doing, we all learn and improve.

[1] Ed Catmull was President and CTO of Pixar who developed the RenderMan rendering system used in films such as Toy Story and Finding Nemo.  He is now President of Walt Disney and Pixar Animation Studios.

------------------------------
Christopher Tang
Professor
University of California-Los Angeles
Los Angeles CA
------------------------------