INFORMS Open Forum

  • 1.  How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-06-2023 13:07

    I'm curious about the current prevalence of Markov Decision Processes (MDPs) in medical decision-making. I've seen several research papers on this topic but was unable to find evidence of real-world implementations. Could you share examples of the widespread use of MDPs in the medical field (preferably with references)?

    Thanks!



    ------------------------------
    Seyed Amirhossein Moosavi
    University of Ottawa
    Ottawa ON
    ------------------------------


  • 2.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-07-2023 10:03

    Seyed -- I recommend that you reach out to experts who first worked together at University of Pittsburgh -- Andrew Schaefer at Rice University and Oguzhan Alagoz at University of Wisconsin.  They would be the experts in applications of MDP in MDM (i.e., Markov Decision Processes in medical decision making).  Good luck! - Scott



    ------------------------------
    Scott Cantor
    Professor
    The University of Texas MD Anderson Cancer Center
    Houston TX
    ------------------------------

    Original Message


  • 3.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-07-2023 10:05

    Indeed, I have a good amount of work using MDPs and being applied to healthcare, however, I am very much on the theoretical side; so while I can point you to hundreds of MDP papers in healthcare, I am not sure how many are actually implemented in practice. Notice that often the result of an MDP paper is a policy recommendation rather than something that gets directly implemented. 

    I could perhaps direct you to a better person to ask: I know Margaret Brandeau at Stanford heavily uses MDPs and has been very influential on healthcare practice.



    ------------------------------
    Sasa Zorc
    Assistant Professor
    University of Virginia, Darden School of Business
    Charlottesville VA
    ------------------------------

    Original Message


  • 4.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-07-2023 11:32

    MDPs for medical decision making have been around for a very long time and are very wide spread.

     Here is a "practical guide" from 1993: https://journals.sagepub.com/doi/abs/10.1177/0272989X9301300409

    As well as a tutorial from 2009: https://journals.sagepub.com/doi/abs/10.1177/0272989X09353194

     The person asking the question is right though that it's hard to know what is actually implemented, unfortunately that is not often documented.  However, I do know that MDPs are used often by companies that consult with pharma and do cost-effectiveness analyses. Though I think what is used more often is evaluations of Markov Chains to test different strategies, not necessarily MDPs. 



    ------------------------------
    Maria Mayorga
    Professor
    North Carolina State University
    memayorg@ncsu.edu
    ------------------------------

    Original Message


  • 5.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-07-2023 11:42

    1. "I'm curious about the current prevalence of Markov Decision Processes (MDPs) in medical decision-making"

    Answer: I would say quite prevalent. They have been widely applied to several health care domains such as organ transplantation, disease screening, chronic treatment initialization and selection, infectious disease modeling, patient discharging decisions,  etc. Andew Schaefer published and earlier review paper on MDP applications, and after that there were at least two more literature reviews considering MDP applications in medical decision making.  

    2. "I've seen several research papers on this topic but was unable to find evidence of real-world implementations."

    Answer: Now the answer here would vary from a few examples to many examples depending on what do you mean with real-world implementations? 

    a. If you mean realistic analysis with clinical data, most of the examples I listed above would go into this category. Most of the time, MDP applications in medical decision making are developed by considering its clinical relevance and their utility to clinicians. 

    b. If you mean MDP analysis providing insights for or inspiring changes or discussions around such changes, the answer would be a good number of such examples are available. For example, our work with Oguzhan Alagoz on a POMDP application on colorectal cancer screening has identified particular patterns in ideal timing of colonoscopy screening. Then in an follow-up paper in the journal of Cancer, we analyzed these patterns for the surveillance of CRC survivors in a more practical setting with a simulation. That follow-up paper got later cited as evidence in the updated colorectal surveillance guidelines. Similarly, Oguzhan Alagoz has been using his MDP-based models providing insights for policy changes in breast cancer and organ transplantation, and his works with the CISNET groups eventually led to policy changes if I do not recall wrong.

    c. If you mean, "an MDP being applied to real-time medical decision making" then the answer would be few. The main reason is in two folds a) many clinicians still consider MDPs as black boxes and would not be feel comfortable in relying on them completely, b) MDPs provide very dynamic and complex policies that could be hard to implement in real-life (that is why they are often used for generating insights for changes rather than generating the changes themselves). Exceptions could be situations where MDM process is already considered complex and computerized. For example, some of the recent works estimating COVID-19 propagation used MDP components and those models were used by state officials, plus a friend of mine has an MDP model (under review probably) on Sepsis, which I believe has been implemented at least partially in the hospital he was collaborating.       



    ------------------------------
    Fatih Safa Erenay,
    Department of Management Sciences University of Waterloo
    Associate Professor
    200 University Ave. CPH 4323,
    Waterloo ON N2L 3G1
    Phone: 519 888 4567 (32521)
    ferenay@connect.uwaterloo.ca
    https://uwaterloo.ca/scholar/ferenay
    ------------------------------

    Original Message


  • 6.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-07-2023 12:09

    Helpful replies have been provided from others.  Chiming in to add that one way that MDP policies could be implemented is by sharing the policies through a web-based tool.  We plan to do this as part of our recently funded 4-year NIH study using a MDP model and related models to guide treatment decisions for kidney cancer patients.  https://reporter.nih.gov/search/42BNVEztI0WguPOwq98YEA/project-details/10734606

    Note, any websites posting nomograms, MDP policies, etc., would need to have proper disclaimers such as those listed on this site: https://www.mskcc.org/nomograms



    ------------------------------
    Jennifer Lobo
    Associate Professor
    University of Virginia
    Charlottesville VA
    ------------------------------

    Original Message


  • 7.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-08-2023 20:44

    Seyed - As I responded on LinkedIn...

    o Every "Markov decision process" is a sequential decision problem, but when people refer to MDPs in the OR community, they typically assume you are going to use Bellman's equation.  I am sure there are countless papers using Bellman's equation in the academic literature, but as you suspect, I also suspect that there are very few real-world uses of Bellman's equation in practice, and there are a variety of reasons for this.  One of the most important is the lack of transparency (a big issue in health).

    o Doctors follow "protocols" which is the word they use for "policy".  They don't design the protocols - those come from the medical research community and governing boards (I am not a health expert - I am not exactly sure of the process for designing protocols, but it is not the doctors).

    o I do not know how the protocols are designed, but they have to use some method.  I doubt that it is with Bellman's equation, but I can guarantee that it is one of my four classes of policies (see https://tinyurl.com/sdalinks for a wealth of information on this).  The medical community needs transparency in their protocols, which Bellman's equation lacks.  I am pretty sure that the process of designing the protocols can be described as a PFA or CFA (see the link above), but these imply tunable parameters, and I am not at all sure that the parameter tuning is being done in a formal way (but I honestly do not know the answer to this).  

    o Note that protocols are often designed as an output of a clinical trial, but not always.  Some drugs are approved for one application, and then doctors discover other applications ("off label").  When this occurs, the protocol is being designed outside of a clinical trial (again - I am out of my expertise here).

    I don't want to make any sweeping statements about how decisions are made in health - this is a deep and rich field with a lot of smart people, but I suspect that relatively few of the more sophisticated methods in the academic literature are used in practice.  Don't let this discourage you - it is an opportunity.  At the same time, rather than barging in to a field like health telling them they are doing it wrong, you need to look for areas where they are asking for help.  My experience is that medical decision making may be difficult to crack.  

    An easier path into problem domains like health is to start from more strategic questions rather than trying to change how real decisions are made in the field.  For example, you might want to estimate the level of adoption of a new COVID vaccine that requires simulating the decisions that doctor's make (rather than actually showing up and trying to change how they make decisions).  ALthough a doctor might not use a sophisticated tool, you may still need a reasonably sophisticated method to mimic the intelligence of how doctor's make decisions (this is something we do at Optimal Dynamics, the company that I started where we optimize driver dispatching decisions in a simulator to help with other decisions).  

    Health is a huge area - spend some time in it and talk to experts who are familiar with how problems are solved to find the areas where people actually need help.



    ------------------------------
    Warren Powell
    Chief Innovation Officer, Optimal Dynamics
    Professor Emeritus, Princeton University
    http://www.castle.princeton.edu/
    ------------------------------

    Original Message


  • 8.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-11-2023 00:08

    Many solid replies have been posted by others.  I want to highlight and drill a bit into one scenario where medical research and Markov processes come together and positively impact healthcare delivery.  At Stanford School of Medicine's Clinical Excellence Resesarch Center (CERC) and other research centers around the globe, multi-disciplinary teams design innovative healthcare delivery methods striving to achieve the triple or quadruple aim (better clinical outcomes, better patient experience, lower healthcare spend, and improved provider experience).  In chronic disease management, for example, clinical guidelines identify stages of disease with associated clinical protocols and costs.  Markov chains effectively model salient aspects of chronic disease progression from stage to stage.

    Clinical protocols published in the medical literature may quantify their impacts on disease progression rates and average healthcare spends, oftentimes using statistically derived point estimates and confidence intervals, yet published data often embodies significant uncertainty.  Such uncertainty poses an additional  barrier to adoption of medical innovations among evidence-based communities of physician specialists (Oncologists, Cardiologists, Nephrologists, etc.).   This is in addition to the "Markov chains are a black box" effect described by another respondant. 

    Healthcare delivery innovations, once designed and proposed, undergo testing in controlled settings prior to widespread adoption.   A key step involves gaining the support of early adopter providers and payors, who must decide whether or not to adopt (providers) and pay for (payors) a healthcare delivery innovation (with possible adaptations) based on possibly scant initial clinical evidence and forecasted cost-effectiveness analysis.

    https://pubsonline.informs.org/doi/abs/10.1287/opre.2017.1685 

    The above link connects to a published paper in Operations Research describing an effective methodology for resolviong data uncertainty in Markov chains with applications to cost-effectiveness analysis of medical innovations.  This paper analyses FIT testing, an alternative to colonoscopy for early colon cancer screening.  The FIT testing protocol and the analysis published in this paper were developed at CERC during 2011-2013.  While not claiming any direct connection, you may have noticed recent FIT testing advertisements (in those commercials with people singing like Sinatra, "...and I did it my way...").  FIT testing is becoming more widely adopted, albiet now ten years after the initial work at CERC (and elsewhere).

    David Moore

    Careflow Logic, LLC

    Portland, OR



    ------------------------------
    David Moore
    Owner
    Careflow Logic, LLC
    Oregon City OR
    ------------------------------

    Original Message


  • 9.  RE: How Widespread Is the Application of MDPs in Medical Decision Making?

    Posted 12-11-2023 13:32

    Dear all,

    Sorry to be late - great question!

    My colleagues are implementing our POMDP policies as described in the following paper:

    https://acsjournals.onlinelibrary.wiley.com/doi/pdf/10.1002/cncr.32597

    This is happening at MD Anderson in Houston.

    Cheers,

    Andrew



    ------------------------------
    Andrew Schaefer
    Professor
    Rice University
    Houston TX
    ------------------------------

    Original Message


Related Content