INFORMS Open Forum

BALTIMORE, MD, April 09, 2024 – Do market researchers still need to conduct original research using human participants in their work? Not always, according to a new study, which found that thanks to the increasing sophistication of large language models (LLMs), human participants can be substituted with LLMs and still generate similar outputs as those generated from human surveys.

Click here to read the full press release.

------------------------------
Ashley Smith
Public Relations Specialist
INFORMS Public Affairs Coordinator
Baltimore MD
------------------------------

Mostly commenting here so that I can get updates on this conversation.  There is certainly cost savings to be gained, although I suspect there are some ethical pitfalls lying in wait using this approach.  I might now have to go play around with some LLMs again, to see what trouble I can get into ;-)

------------------------------
Nicholas Ulmer
CANA
Pacific Grove CA
------------------------------

Interesting, and the idea seems obvious, once you hear it. As someone with ZERO knowledge of marketing, two thoughts occur immediately:

1. Suppose a market survey is interested not just in the most likely sentiment, but also the diversity of sentiments. Can an LLM replicate that? What if I am interested in the "fringe" element that does not contribute to the corpus upon which LLMs are built? Suppose I engineer prompts designed to elicit sentiments from different market segments; will LLMs have very different fidelity on different segments? 
2. If the results of an LLM market survey become part of the corpus upon which LLMs are later trained do we slowly move away from what people think and toward what LLMs think?

------------------------------
Barry Nelson
Walter P. Murphy Professor
Northwestern University
Evanston IL
------------------------------

Hi Barry, I think these are great questions! Here is a recent paper on "psychologically-valid generative agents", which is not directly on this capability, but certainly relevant: https://ojs.aaai.org/index.php/AAAI-SS/article/view/27698

------------------------------
Esma Gel
Cynthia Hardin Milligan Chair of Business and Professor
University of Nebraska-Lincoln
Lincoln NE
------------------------------

Original Message:
Sent: 04-20-2024 10:17
From: Barry Nelson
Subject: Can Large Language Models Replace Human Participants in Some Future Market Research?

Interesting, and the idea seems obvious, once you hear it. As someone with ZERO knowledge of marketing, two thoughts occur immediately:

1. Suppose a market survey is interested not just in the most likely sentiment, but also the diversity of sentiments. Can an LLM replicate that? What if I am interested in the "fringe" element that does not contribute to the corpus upon which LLMs are built? Suppose I engineer prompts designed to elicit sentiments from different market segments; will LLMs have very different fidelity on different segments? 
2. If the results of an LLM market survey become part of the corpus upon which LLMs are later trained do we slowly move away from what people think and toward what LLMs think?

------------------------------
Barry Nelson
Walter P. Murphy Professor
Northwestern University
Evanston IL
------------------------------