INFORMS Open Forum

I agree with Barry's perspective, and want to add 3 thoughts:

1. He said that "We should support and believe in editing." I think he refers, primarily, to the Associate Editor (AE) role and higher editorial roles. Part of the AE's job is to delineate the changes required in a revision after reflecting on referee reports and the AE's own opinions of the paper. The list should not be the union of the issues raised in the referee reports; we, as referees, seem to come up with endless additional recommended work of little value for we, the submitting authors! AEs and higher-level editors need to act against that trend to avoid what I'll call "referee bloat" in papers, to give a name to Barry's concern. On that note, I admire the initiative begun by Chris Tang while he was Editor-in-Chief of MSOM, which has retained some momentum in MSOM and other journals; see, e.g., this post from Chris.
2. We all complain about "the referees." But we are the referees! Let's try to provide the reports that we would like to receive, rather than reports reflecting our most traumatic recent experience with our own paper submission. Reflecting our own trauma, even if it is unconscious, can only lead to a downward spiral to doom!
3. It may be difficult to avoid "defensive writing" (I heard this phrase from David Simchi-Levi to describe writing that anticipates and attempts to head off referee concerns) in papers, because of the current equilibrium in which we find ourselves, but it is worth a try for the reasons Barry advanced.

To those referees and editors who subscribe to Barry's perspective, thank you! To those who do not or haven't thought about it, please reflect on it!

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org
------------------------------

Thanks for raising the topic and for the insightful comments. As a junior researcher in our field, I also agree with these points.

I just want to share two thoughts that I had when I review the papers (sorry if it's a bit detour from the main theme of the discussions here):

1. Can we introduce a confidence score system to indicate the confidence of a submitted review? Generally, as reviewers, we may have different confidence levels for different papers we review, depending on the familiarity with the topic and the time spent on the review. 
2. Can we make all the reviews public in an anonymous manner (which may or may not depend on the willingness of the reviewers)? That's how it works for many CS conferences on the platform such as Openreview (https://openreview.net/). The critiques in the reviews can provide a guidance for readers, and in particular, for graduate students and junior researchers in our community to get a better idea of how to produce good research and write good papers. One step further, this type of Openreview platform also allows the general audience (who are not the authors or the reviewers of the paper) to comment on a paper anonymously; and this could facilitate discussions of a paper and exchange of ideas.

This might not be an easy change, even from the technical aspect of the submission system, but I'd be happy to see them in the near future. 

------------

Xiaocheng Li

Assistant professor of Analytics and Operations

Imperial college business school, Imperial College London

https://xiaocheng-li.github.io/ 

Original Message:
Sent: 07-25-2024 12:17
From: Bruce Hartman
Subject: Is our publication model an existential threat to our field?

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org

Thanks for raising the topic and for the insightful comments. As a junior researcher in our field, I also agree with these points.

I just want to share two thoughts that I had when I review the papers (sorry if it's a bit detour from the main theme of the discussions here):

1. Can we introduce a confidence score system to indicate the confidence of a submitted review? Generally, as reviewers, we may have different confidence levels for different papers we review, depending on the familiarity with the topic and the time spent on the review. 
2. Can we make all the reviews public in an anonymous manner (which may or may not depend on the willingness of the reviewers)? That's how it works for many CS conferences on the platform such as Openreview (https://openreview.net/). The critiques in the reviews can provide a guidance for readers, and in particular, for graduate students and junior researchers in our community to get a better idea of how to produce good research and write good papers. One step further, this type of Openreview platform also allows the general audience (who are not the authors or the reviewers of the paper) to comment on a paper anonymously; and this could facilitate discussions of a paper and exchange of ideas.

This might not be an easy change, even from the technical aspect of the submission system, but I'd be happy to see them in the near future. 

------------

Xiaocheng Li

Assistant professor of Analytics and Operations

Imperial college business school, Imperial College London

https://xiaocheng-li.github.io/ 

Original Message:
Sent: 07-25-2024 12:17
From: Bruce Hartman
Subject: Is our publication model an existential threat to our field?

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org

As an Author, AE, and referee I support Xiaocheng's recommendations. I also agree that rapid online publishing will not be as reliable as the old-fashioned double-blind system. However, I have to highlight one subtle observation: unfortunately, after some point, many scientists who benefitted from this reliable (double-blind ) system stopped thinking sustainably and started shooting their own feet by acting insecure and selfish (I am not talking about being picky or perfectionist): Many of them thought they could use their obtained power to prevent newcomers from entering their field and hence they can solidify their places. Maybe that was true for some time. However, after some point, they realized that finding good referees had become significantly hard and the decisions (of the Journals) started to be made randomly or politically. Hence they have started digging the grave of their science environment. 

Btw. I am very happy to see this type of straight-to-the-point discussion being held to raise awareness and create some real change. I think feeling responsible and giving back to all sorts of society we belong to is the key.

Warm regards

(Newly resigned faculty due to gender discrimination)

Thanks for raising the topic and for the insightful comments. As a junior researcher in our field, I also agree with these points.

I just want to share two thoughts that I had when I review the papers (sorry if it's a bit detour from the main theme of the discussions here):

1. Can we introduce a confidence score system to indicate the confidence of a submitted review? Generally, as reviewers, we may have different confidence levels for different papers we review, depending on the familiarity with the topic and the time spent on the review. 
2. Can we make all the reviews public in an anonymous manner (which may or may not depend on the willingness of the reviewers)? That's how it works for many CS conferences on the platform such as Openreview (https://openreview.net/). The critiques in the reviews can provide a guidance for readers, and in particular, for graduate students and junior researchers in our community to get a better idea of how to produce good research and write good papers. One step further, this type of Openreview platform also allows the general audience (who are not the authors or the reviewers of the paper) to comment on a paper anonymously; and this could facilitate discussions of a paper and exchange of ideas.

This might not be an easy change, even from the technical aspect of the submission system, but I'd be happy to see them in the near future. 

------------

Xiaocheng Li

Assistant professor of Analytics and Operations

Imperial college business school, Imperial College London

https://xiaocheng-li.github.io/ 

Original Message:
Sent: 07-25-2024 12:17
From: Bruce Hartman
Subject: Is our publication model an existential threat to our field?

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org

As an Author, AE, and referee I support Xiaocheng's recommendations. I also agree that rapid online publishing will not be as reliable as the old-fashioned double-blind system. However, I have to highlight one subtle observation: unfortunately, after some point, many scientists who benefitted from this reliable (double-blind ) system stopped thinking sustainably and started shooting their own feet by acting insecure and selfish (I am not talking about being picky or perfectionist): Many of them thought they could use their obtained power to prevent newcomers from entering their field and hence they can solidify their places. Maybe that was true for some time. However, after some point, they realized that finding good referees had become significantly hard and the decisions (of the Journals) started to be made randomly or politically. Hence they have started digging the grave of their science environment. 

Btw. I am very happy to see this type of straight-to-the-point discussion being held to raise awareness and create some real change. I think feeling responsible and giving back to all sorts of society we belong to is the key.

Warm regards

(Newly resigned faculty due to gender discrimination)

Thanks for raising the topic and for the insightful comments. As a junior researcher in our field, I also agree with these points.

I just want to share two thoughts that I had when I review the papers (sorry if it's a bit detour from the main theme of the discussions here):

1. Can we introduce a confidence score system to indicate the confidence of a submitted review? Generally, as reviewers, we may have different confidence levels for different papers we review, depending on the familiarity with the topic and the time spent on the review. 
2. Can we make all the reviews public in an anonymous manner (which may or may not depend on the willingness of the reviewers)? That's how it works for many CS conferences on the platform such as Openreview (https://openreview.net/). The critiques in the reviews can provide a guidance for readers, and in particular, for graduate students and junior researchers in our community to get a better idea of how to produce good research and write good papers. One step further, this type of Openreview platform also allows the general audience (who are not the authors or the reviewers of the paper) to comment on a paper anonymously; and this could facilitate discussions of a paper and exchange of ideas.

This might not be an easy change, even from the technical aspect of the submission system, but I'd be happy to see them in the near future. 

------------

Xiaocheng Li

Assistant professor of Analytics and Operations

Imperial college business school, Imperial College London

https://xiaocheng-li.github.io/ 

Original Message:
Sent: 07-25-2024 12:17
From: Bruce Hartman
Subject: Is our publication model an existential threat to our field?

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org

 This news about your "resignation" is infuriating.  Perhaps there is something INFORMS can do.  

As an Author, AE, and referee I support Xiaocheng's recommendations. I also agree that rapid online publishing will not be as reliable as the old-fashioned double-blind system. However, I have to highlight one subtle observation: unfortunately, after some point, many scientists who benefitted from this reliable (double-blind ) system stopped thinking sustainably and started shooting their own feet by acting insecure and selfish (I am not talking about being picky or perfectionist): Many of them thought they could use their obtained power to prevent newcomers from entering their field and hence they can solidify their places. Maybe that was true for some time. However, after some point, they realized that finding good referees had become significantly hard and the decisions (of the Journals) started to be made randomly or politically. Hence they have started digging the grave of their science environment. 

Btw. I am very happy to see this type of straight-to-the-point discussion being held to raise awareness and create some real change. I think feeling responsible and giving back to all sorts of society we belong to is the key.

Warm regards

(Newly resigned faculty due to gender discrimination)

Thanks for raising the topic and for the insightful comments. As a junior researcher in our field, I also agree with these points.

I just want to share two thoughts that I had when I review the papers (sorry if it's a bit detour from the main theme of the discussions here):

1. Can we introduce a confidence score system to indicate the confidence of a submitted review? Generally, as reviewers, we may have different confidence levels for different papers we review, depending on the familiarity with the topic and the time spent on the review. 
2. Can we make all the reviews public in an anonymous manner (which may or may not depend on the willingness of the reviewers)? That's how it works for many CS conferences on the platform such as Openreview (https://openreview.net/). The critiques in the reviews can provide a guidance for readers, and in particular, for graduate students and junior researchers in our community to get a better idea of how to produce good research and write good papers. One step further, this type of Openreview platform also allows the general audience (who are not the authors or the reviewers of the paper) to comment on a paper anonymously; and this could facilitate discussions of a paper and exchange of ideas.

This might not be an easy change, even from the technical aspect of the submission system, but I'd be happy to see them in the near future. 

------------

Xiaocheng Li

Assistant professor of Analytics and Operations

Imperial college business school, Imperial College London

https://xiaocheng-li.github.io/ 

Original Message:
Sent: 07-25-2024 12:17
From: Bruce Hartman
Subject: Is our publication model an existential threat to our field?

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org

Thanks for raising the topic and for the insightful comments. As a junior researcher in our field, I also agree with these points.

I just want to share two thoughts that I had when I review the papers (sorry if it's a bit detour from the main theme of the discussions here):

1. Can we introduce a confidence score system to indicate the confidence of a submitted review? Generally, as reviewers, we may have different confidence levels for different papers we review, depending on the familiarity with the topic and the time spent on the review. 
2. Can we make all the reviews public in an anonymous manner (which may or may not depend on the willingness of the reviewers)? That's how it works for many CS conferences on the platform such as Openreview (https://openreview.net/). The critiques in the reviews can provide a guidance for readers, and in particular, for graduate students and junior researchers in our community to get a better idea of how to produce good research and write good papers. One step further, this type of Openreview platform also allows the general audience (who are not the authors or the reviewers of the paper) to comment on a paper anonymously; and this could facilitate discussions of a paper and exchange of ideas.

This might not be an easy change, even from the technical aspect of the submission system, but I'd be happy to see them in the near future. 

------------

Xiaocheng Li

Assistant professor of Analytics and Operations

Imperial college business school, Imperial College London

https://xiaocheng-li.github.io/ 

Original Message:
Sent: 07-25-2024 12:17
From: Bruce Hartman
Subject: Is our publication model an existential threat to our field?

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

------------------------------
Bruce Hartman
Professor
University of St. Francis
Tucson, AZ United States
bruce@ahartman.net
website:http://drbrucehartman.net/brucewebsite/
blog:http://supplychainandlogistics.org

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

Excellent points by Barry, Shane, and Bruce!

Looking at the big picture, much of our scholarly publishing model is built upon the pre-digital world, where hardcopy articles were expensive to print and distribute. If we were designing a scholarly publishing model from scratch for our digital world, what would it look like? Maybe something like...

Authors publish whatever they want. Rather than the work being examined by a small set of referees, everybody has access to it. Comments and improvement suggestions come from anyone and from anywhere. Eventually, the best work becomes recognized, widely respected, and influential. AI helps with beautifying the raw papers from authors.

Those are just a few half-baked ideas for how this may work. The main point is to flip the scholarly publishing model upside down. Rather than delays from referees, get the work published instantly. Have the refereeing/feedback/improvements come after the work is visible. As Barry noted, important concepts like Schruben's 1983 work impact the world before being buried with incremental improvements.

I agree with Barry, and with Shane.  One pet peeve of mine is the endless literature reviews in articles today.  Just because you looked it up is no reason to put it in the paper. Only the articles directly relevant to your subject should be included and summarized.  That might be three or four or five, unless the submitted article has very wide scope.

I also agree about endless proofs, some of them trivial, in appendices.  Refining your proofs may take longer in writing, but when the reader has to go to the appendix for substantiation, it wrecks the reading flow.

And Shane, thanks for providing the link to Chris Tang's article.  Limiting what reviewers say to main points is very helpful.  I've read too many 'response to reviewers' letters from authors, detailing every small point authors felt they had to respond to.

It's also helpful if reviewers think hard about the subject and don't make comments that are just plain wrong, but need to be rebutted. More openness to  new ideas would also be useful, as we run the risk of ever refining a few ideas that were originally good, rather than publishing genuinely new ideas.  My advisor Moshe Dror, God rest his soul, frequently railed about 'epsilon' papers that contributed just that much to a topic.

I don't agree 100% with this thread.  As an AE, referee and author -- I think our behavior is completely rational given the pressures to publish and the trend towards deep specialization.   The biggest contributor to friction, as noted in this thread, is the publication review. It is not possible for a single individual to be aware of every relevant paper.  But it does seem to be the expectation.  

Perhaps we can consider how technology innovations might help.  Rapid publication online is now the default in many fields, and there are many venues: Arxiv, Optimization Online, SSRN, etc.  Large Language Models are improving at an astounding pace.  Can these be helpful?

What if every submission was immediately posted online in a venue that supports comments?  What if every such posting automatically generated a summary of the contribution and links to possibly relevant publications?  What if every comment was automatically processed to detect relevance and quality?  

Would this replace scholarly review or threaten the business models of the publishers?  I don't think that it does.  Editorial boards can decide to reject papers that cite online sources.  

I agree with all the points above, and I am glad to see a discussion like this getting traction! What we have achieved is indeed due to rational behavior, which makes me wonder if we can change the underlying mechanisms to prioritize communication and development of new ideas.

Like Xiaocheng, I have seen some interesting ideas in CS conferences that might be worth considering:

• Prizes for top reviewers and AEs, such as the top 10% at a given year according to feedback from AEs and editors. ICML uses that percentage.
• A strict page limit encouraging authors to either be succinct, or to at least restrict the content of the paper to their main ideas and save the rest to the appendix. Over time, that led to interesting new styles, such as papers containing only the most relevant theorems and an idea of how these results were proved.
• The confidence scores are indeed a helpful guide, since reviewers self-declare how comfortable they are with the content of the paper, and in doing so it becomes clear which reviewers should have greater influence on the final decision. On that note, there is usually a discussion period among reviewers in which AEs (actually ACs in CS terms) typically ask reviewers to come to an agreement about the paper. I wonder if something like that could be achieved by automatically sending all reviews to all reviewers and giving them a week to optionally comment on the overall evaluation before the AE acts.

As a new AE, I will also try to have in mind the points above about being critical about which reviewers comments should be more seriously followed by the authors. I appreciate the advice.

I don't agree 100% with this thread.  As an AE, referee and author -- I think our behavior is completely rational given the pressures to publish and the trend towards deep specialization.   The biggest contributor to friction, as noted in this thread, is the publication review. It is not possible for a single individual to be aware of every relevant paper.  But it does seem to be the expectation.  

Perhaps we can consider how technology innovations might help.  Rapid publication online is now the default in many fields, and there are many venues: Arxiv, Optimization Online, SSRN, etc.  Large Language Models are improving at an astounding pace.  Can these be helpful?

What if every submission was immediately posted online in a venue that supports comments?  What if every such posting automatically generated a summary of the contribution and links to possibly relevant publications?  What if every comment was automatically processed to detect relevance and quality?  

Would this replace scholarly review or threaten the business models of the publishers?  I don't think that it does.  Editorial boards can decide to reject papers that cite online sources.  

I agree with all the points above, and I am glad to see a discussion like this getting traction! What we have achieved is indeed due to rational behavior, which makes me wonder if we can change the underlying mechanisms to prioritize communication and development of new ideas.

Like Xiaocheng, I have seen some interesting ideas in CS conferences that might be worth considering:

• Prizes for top reviewers and AEs, such as the top 10% at a given year according to feedback from AEs and editors. ICML uses that percentage.
• A strict page limit encouraging authors to either be succinct, or to at least restrict the content of the paper to their main ideas and save the rest to the appendix. Over time, that led to interesting new styles, such as papers containing only the most relevant theorems and an idea of how these results were proved.
• The confidence scores are indeed a helpful guide, since reviewers self-declare how comfortable they are with the content of the paper, and in doing so it becomes clear which reviewers should have greater influence on the final decision. On that note, there is usually a discussion period among reviewers in which AEs (actually ACs in CS terms) typically ask reviewers to come to an agreement about the paper. I wonder if something like that could be achieved by automatically sending all reviews to all reviewers and giving them a week to optionally comment on the overall evaluation before the AE acts.

As a new AE, I will also try to have in mind the points above about being critical about which reviewers comments should be more seriously followed by the authors. I appreciate the advice.

I don't agree 100% with this thread.  As an AE, referee and author -- I think our behavior is completely rational given the pressures to publish and the trend towards deep specialization.   The biggest contributor to friction, as noted in this thread, is the publication review. It is not possible for a single individual to be aware of every relevant paper.  But it does seem to be the expectation.  

Perhaps we can consider how technology innovations might help.  Rapid publication online is now the default in many fields, and there are many venues: Arxiv, Optimization Online, SSRN, etc.  Large Language Models are improving at an astounding pace.  Can these be helpful?

What if every submission was immediately posted online in a venue that supports comments?  What if every such posting automatically generated a summary of the contribution and links to possibly relevant publications?  What if every comment was automatically processed to detect relevance and quality?  

Would this replace scholarly review or threaten the business models of the publishers?  I don't think that it does.  Editorial boards can decide to reject papers that cite online sources.  

Dear Barry, thank you so much for sharing your thoughts.  I would like to thank those who responded to your concerns.  

First of all, as an author, I have expressed my concerns about the length of the paper, and endless extensions and robustness checks.  I have suggested that we should focus on key ideas and the base model, and leave all other extensions and robustness checks in the appendix so that the paper is clean and readable.   But the problem lies with us -- many reviewers, AEs, and DEs ask for endless things.  I have written a piece based on my personal journey -- https://pubsonline.informs.org/do/10.1287/orms.2022.01.01/full/

I am indebted to the INFORMS community which has supported me over the last 4 decades and I am committed to making changes.  I have worked with Matt Walls to make changes, but there are many obstacles.   But I shall continue to push for change.  

I hear you, I share the same concerns, and I support all the ideas.  Regarding reviewer recognition, it is coming soon. 

In terms of getting reviewers, AEs, and DEs to do their jobs properly, and getting the authors to do their jobs properly, we need to think of a way to improve our co-production process.   No review process is perfect, but we need to find a way to improve.   I think the EIC can play an important role in leading changes.   I shall continue to ask EICs for their help and support.   

How to improve the review process?  This question has been in my mind over the last 20 years.  See my OR/MS today article -- we are all volunteers, how do we encourage, motivate, and incentivize everyone to do their jobs properly?   

I chaired a task force 4 years ago, and it appears that EICs are the ones who can make changes.  I had the privilege to make changes when I served as EIC of MSOM.   I think all EICs can make changes, and I shall continue to urge them to take on leadership roles to lead changes. 

Meanwhile, I welcome your suggestions.  

Chris Tang

VP (pubs)

------------------------------
Christopher Tang
Distinguished Professor
ucla
Los Angeles CA
------------------------------

Thank you for Barry's insights on the issue. 

Thank you for Barry's insights on the issue.