Can AI really mimic our decision-making?

When given clear context and guided reasoning, AI can mirror human strategic decisions

Featured faculty: Philip Zhang Renyu and Jenny Jin
Written by Joanne Madrid

Large language models or LLMs, the AI systems behind chatbots like ChatGPT, are increasingly being used as stand-ins for humans in behavioural research. Dubbed synthetic surveys or silicon sampling, people can now ask AI to complete surveys at a fraction of the cost and time of human participants.

AI startups like Simile, Aaru, Artificial Societies, Uxia, and many others offer human behaviour simulations to predict market trends, consumer preferences, and even political views. This synthetic survey is a direct application of social simulation, in which LLMs act as humans in experimental settings. As LLM research expands, an important question is whether large language models can accurately reproduce human decisions in social experiments.

Yet there is a catch. Social simulation may not be a reliable substitute for human respondents because AI can struggle to capture nuanced opinions and may reflect biases in its training data. Although large language models have successfully reproduced simple experiments, they often fail in complex scenarios. This challenge has become an important area of LLM research, particularly in understanding why AI reasoning sometimes differs from human decision-making.

According to Philip Zhang Renyu, Professor in the Department of Decisions, Operations and Technology at the Chinese University of Hong Kong (CUHK) Business School, such limits may stem from a deeper structural challenge in how LLMs understand and reason about their tasks.

“LLMs in behavioural and social simulations can fail not only because of flawed reasoning, but also because of the limitations in how they interpret their tasks,” he says. “Even if they understand the tasks, their reasoning process may not align with human thinking.”

Professor Zhang’s latest study, Improving behavioural alignment in LLM social simulations via context formation and navigation, co-authored with Jenny Jin, Assistant Professor in the same department and their PhD student Kong Letian, proposes a two-stage method to improve LLMs’ ability to simulate human behaviour.

Beyond clever prompts

Most AI users engaged in prompt engineering have already recognised the importance of clear, specific instructions, but many popular prompts still fall short in social simulations. Therefore, the study aims to find whether a more comprehensive approach is more effective.

Specifically, Professor Zhang and the team turn to a classic Human Problem-Solving Theory developed by Herbert Simon and Allen Newell. Both celebrated scientists, with the help of programmer Cliff Shaw, developed Logic Theorist, the first computer programme designed to perform automated reasoning that marked the birth of AI.

Simon and Newell argue that humans solve problems by forming a mental picture of the task and selectively searching for solutions within it. Inspired by this theory, the team then develops a two-stage framework to better align LLMs with human behaviours in social simulations.

The first stage is context formation, or explicitly specifying experimental design so LLMs can understand the tasks and their background by defining their goals, what information matters, and how all the pieces fit together. The second is context navigation, or guiding the reasoning process so LLMs can further strategise, update their assessments, and weigh the best options.

For LLMs to behave like humans, Professor Zhang notes that the two-stage framework, context formation and context navigation, should be clearly defined so the AI understands the task and its role perfectly.

“Even really smart AI still needs a crystal-clear instruction to make good decisions. While AI sometimes can reason autonomously in simpler tasks, context navigation remains crucial in complex scenarios to choose the most relevant behavioural signals.”

Professor Philip Zhang Renyu

Comparison with popular prompts

Current popular prompting techniques can improve task performance but are not always suitable for social simulations. For instance, chain-of-thought prompting focuses on step-by-step instructions, yet if the AI misunderstands the rules and setup, it often yields inconsistent results. Few-shot prompting guides the AI model with examples, but it may not be reliable when the social experiment differs from the examples.

Tree-of-thought prompting asks the AI model to explore several paths to choose the best solution, but it fails to determine which one is most human-like. Meta-prompting can organise reasoning by designing a reusable prompt, but still requires humans to specify the setting and the standard for a valid result.

“Context formation provides a systematic way to explain the rules and setup of an experiment to the AI models, which helps it understand and adapt to specific scenarios rather than relying on a few examples or generic instructions,” Professor Zhang says.

“Context navigation directs the AI models selectively towards the crucial information and strategic considerations, rather than requiring an open search across reasoning paths. These stages help the AI figure out why it still mismatches human behaviour and see if it misunderstood the setting, if its reasoning was incorrect, or both.”

Putting the method to the test

To evaluate the framework, the team tested it across three existing social experiments with publicly available human data, allowing the researchers to compare AI-generated decisions directly with real human choices. They provided the experimental instructions step by step, as in chain-of-thought prompting, to the most capable LLMs at the time—GPT-4o, GPT-5, Claude 4 Sonnet Thinking, and DeepSeek-R1—to assess how accurately these large language models could replicate human behaviour.

The first human experiment found that the waiting time to get a product can signal its quality. When the same instructions in the experiment were fed to LLMs, all models boiled the problem down to a cost analysis and ignored social cues humans perceived.

Adding context formation that some queuing buyers already knew the product’s quality helped the LLMs understand the instruction, but still could not replicate human behaviour. Only after providing context navigation, that queue length can be a meaningful cue for quality, did the LLMs start to act like human shoppers.

The second experiment asked humans to choose whether to invest in a project without knowing its true quality. Some project owners offered investors a refund if they failed to reach full funding, which human investors viewed as a signal of a good project.

However, even after being provided with investment rules, all LLMs significantly funded fewer good projects than humans. Only when the LLMs received clear context navigation to interpret the refund scheme as a positive signal did they replicate human investors.

Not every situation demanded both steps. The third experiment simply found that human purchases decreased as the price increased. Oddly, upon receiving the instructions, LLMs bought more when the price rose, and as the cost soared further, they eventually reduced their purchases.

When LLMs were given a clear explanation of the experimental setup, they successfully replicated human behaviour. Here, providing context formation was sufficient. The contrast points to a useful rule of thumb: simple choices often need only context formation, while more complex scenarios involving several parties require a two-stage method.

As AI models improve, explicit guidance on how to reason may matter less for simpler tasks since the models can work it out for themselves, but this doesn’t mean that context navigation becomes less important. “Even really smart AI still needs a crystal-clear instruction to make good decisions,” Professor Zhang adds.

“While AI sometimes can reason autonomously in simpler tasks, context navigation remains crucial in complex scenarios to choose the most relevant behavioural signals.”