INFORMS Open Forum

Friends,

The Vice President and CEO of Innovation at Microsoft, Jason Wild, predicted that every job will be transformed by an existence of an AI co-pilot, driven by large language models (LLMs) such as ChatGPT (see Microsoft: Every job will have an artificial intelligence copilot).  Microsoft is already driving such a transformation by infusing ChatGPT technologies into office - see Microsoft co-pilot for work). 

When applying OR/analytics to solve real world problems, there are two fundamental roles: The business professional who makes the decision, and the OR/Analytics professional.

This got me thinking about the following questions: Can co-pilots for decision making, created by combining LLMs and OR techniques, and targeting these two roles, also transform the way decisions are made? How can a co-pilot for decision making provide value for the business professionals making such decisions? Can a co-pilot be created, to, for example, guiding analytics professionals in applying OR by, for example, walking them through the INFORMS CAP methodology in the context of the specific application? What other features should such co-pilots provide? 

For a couple of initial examples of how LLMs can be used in the context of decision making, see some of my experiments in using ChatGPT to create optimization models starting from minute 19 of the video here and ChatGPT Does Decision Intelligence for Net Zero).

Would love to hear your feedback!

------------------------------
Segev Wasserkrug
Research Staff Member
IBM Research - Israel
Haifa
------------------------------

AI requires a test data set to learn from.  It requires lots of occurrences.  The example that comes to mind is being able to identify a cat from a dog or lion.   There are certainly lots of example of this.   ChatGPT being one of them.   But then there are larger decisions that are more unique.   Traditionally these decisions have been made using Subject Matter Expert opinions (and engineering) along with some calibration of historical data (sometimes)   Often the SME assessments are biased toward a good outcome and this bias needs to be accounted for in some manner.  Whether it is economical or not, an AI / ML study could be done along side of the traditional analysis leading to a decision.   Only through these pilot studies will we know whether AI could be used for decisions that tend to be more unique because of fundamental differences.

------------------------------
BRIAN PUTT
Decision Scientist Consultant
retired
Fremont CA
------------------------------

Great question I need to think more about.  One of the most critical components for success of an analytics project are agents of change - where the same person has analytics skills and business skills.  For a description of agents of change see paper by Prof. John Milne, I have summarized part of this in an Arkieva blog.   This person has to be able to read a room in real time seamlessly mixing and matching skills.  Its not the formulation that is the challenge, it is whether it will solve and the hunt for the data.
This is just my initial thought - I need to give this more thought

------------------------------
Ken Fordyce
director analytics without borders
Arkieva
Wilmington DE
------------------------------

Thanks, Segev.  You pose an interesting question that I have also been thinking about.  I think ChatGPT and similar software are enjoying a peak in the "hype cycle".  Typically, that is followed by a period of disillusionment, and finally into steady growth.  So, expect a bunch of failures and negative reports.

But, I also agree with you that many (most?) jobs will be changed as AI becomes an all-pervasive tool.  I like your suggestion of a co-pilot.  Many functions will be completely automated.  AI will provide guidance and insight for the rest.  People sometimes know more about a specific topic than external data sources fed into any particular AI tool, so people can be needed to make the final decision.  Interestingly, in demand forecasting - a common supply chain activity - most studies show that the human input actually adds to Forecast Error, rather than improving the forecast.

So, I would say YES, decision making can definitely benefit from and AI co-pilot. 

------------------------------
Duncan Klett
------------------------------

AI co-pilot options such as the various GPT options being launched almost daily....are definitely disruptive.  Differentiating an analytics professional from software development, I would say their is a reliance on tools and resources to be quickly and efficiently get to a recommendation to a decision maker.  GPT options can definitely support, although not replace, the work of an analytics professional.  Personally I view GPT options as a search engine on steroids.  Not only providing results, but help in moving to a solution faster.  The AP needs to be provide the validation of the feedback from GPT.  The solutions are not perfect, can have bias, and still require the human in the loop.  One might think of an AP using these new tools as an augmented human.  The better either is, the better the combination will be.

------------------------------
Nick Ulmer, CAP
Faculty
Naval Postgraduate School
Pacific Grove CA
------------------------------

Many thanks for the great comments!

Can you perhaps think of concrete examples of where such a co-pilot can help you as an analytics practitioner?

For example, think if we could  ask for a first draft of a presentation that explains the analysis results in terms that would be understandable to the business person who is trying to solve the problem, and is adapted to the specific problem, and have such a first draft ready in minutes.  Wouldn't this be great? Any other examples?

------------------------------
Segev Wasserkrug
Research Staff Member
IBM Research - Israel
Haifa
------------------------------

We have been building OR-DA algorithms for decision support. For me, these are "co-pilots". They are used for decision-making by car dealers in the USA because we tailored the models for this industry (our system is https://www.frogdata.com/).

So one answer to "How can a co-pilot for decision making provide value for the business professionals making such decisions?" is that it provides the decision-maker with the embedded intelligence of an OR-DA practitioner who has learned by immersion in the business context, and that gets updated as we (the system providers) find ways to improve it.

Using LLM for these business decision agents is non-trivial. Think of it this way:
1. It's a known category of problem, and an optimal solution exists. This would use classical OR-DA.
2. It's a new category or an unsolved problem. When used in a business context, it is stated as developing situation awareness (akin to battlefield awareness) and then path-finding in explore/exploit cycles. Again, classical methods provide firm guides for both awareness (including anomaly detection) and experiment-assessment cycles.

If we include LLMs in the decision-support algorithm, the analytics practitioner is still required to build it. That practitioner can use an LLM as a co-pilot. Analytics practitioners acting as Decision Coaches (h/t Dr. Barrager and Dr. Milne) would also have a role to play in helping business decision-makers use the decision co-pilots because of the range of concerns to be handled. Those Decision Coaches could use an LLM as a co-pilot. So there are three kinds of LLM-co-pilots here:
1. Decision Co-Pilot = classical OR-DA co-pilots used by business decision-makers, augmented by LLM only where appropriate.
2. Decision Co-Pilot Maker = LLM to help the Decision Co-Pilot builder (an OR-DA practitioner). Such a Maker can democratize the role of analytics practitioners.
3. Decision-Coach Co-Pilot = LLM to guide the Decision Coach (an OR-DA practitioner) who helps business decision-makers use the Decision Co-Pilot.

Are we getting to the point of having an INFORMS BoK LLM that could serve these three uses?

------------------------------
Rahul Saxena
FrogData.com
------------------------------

Original Message:
Sent: 05-15-2023 04:39
From: Segev Wasserkrug
Subject: Can Decision Making benefit from an AI driven co-pilot?

Many thanks for the great comments!

Can you perhaps think of concrete examples of where such a co-pilot can help you as an analytics practitioner?

For example, think if we could  ask for a first draft of a presentation that explains the analysis results in terms that would be understandable to the business person who is trying to solve the problem, and is adapted to the specific problem, and have such a first draft ready in minutes.  Wouldn't this be great? Any other examples?

------------------------------
Segev Wasserkrug
Research Staff Member
IBM Research - Israel
Haifa
------------------------------

Absolutely! I am using Bing AI (ChatGPT) to generate R code for my analysis and decision-making processes. Bing AI has been my co-pilot since it was beta tested. 

------------------------------
Alberto Aparicio
Data Analyst
Charitable Adult Rides & Services, Inc.
La Mesa CA
------------------------------

Hi everyone, 

Sharing our work at Microsoft on using Large Language Models (such as GPT-4) for supply chain optimization. Our approach is quite general and can be applied to many decision processes.

[2307.03875] Large Language Models for Supply Chain Optimization (arxiv.org)

------------------------------
Konstantina Mellou
Senior Researcher
Microsoft Research
Redmond WA
------------------------------

Many thanks for this interesting reference!

------------------------------
Segev Wasserkrug
Research Staff Member
IBM Research - Israel
Haifa
------------------------------

Original Message:
Sent: 07-19-2023 00:57
From: Konstantina Mellou
Subject: Can Decision Making benefit from an AI driven co-pilot?

Hi everyone, 

Sharing our work at Microsoft on using Large Language Models (such as GPT-4) for supply chain optimization. Our approach is quite general and can be applied to many decision processes.

[2307.03875] Large Language Models for Supply Chain Optimization (arxiv.org)

------------------------------
Konstantina Mellou
Senior Researcher
Microsoft Research
Redmond WA
------------------------------