Advances, challenges in LLM-based social simulation
By CHEN HUASHAN and QU CHENJI / 11-19-2026
 
Researchers can ask LLMs to assume the roles of respondents with specified sociodemographic characteristics and answer questions for a particular study. Image generated by AI
As generative artificial intelligence becomes increasingly integrated into social science research, large language models (LLMs) are reshaping methods of social simulation. Using prompts, researchers can ask LLMs to assume the roles of respondents with specified sociodemographic characteristics and answer questions for a particular study. The resulting simulated responses form “silicon samples,” which can be obtained at lower cost than conventional methods allow. LLMs can also guide the decisions and interactions of simulated agents, helping researchers examine the relationship between individual action and collective behavior. Simulation results can in turn support investigations into the mechanisms underlying behavior and provide material for empirical testing.
 
Generating silicon samples, drawing inferences
Research using silicon samples generally takes two forms, distinguished by how the model is asked to respond: discriminative probing and generative probing. Discriminative probing requires the model to choose from predefined options or respond on a specified scale—for instance, rating personal happiness on a scale from 1 to 4. Generative probing, by contrast, allows the model to express its judgments and reasoning in natural language, such as explaining whether employee bonuses should be based on performance or position. The former facilitates statistical analysis of the distribution of responses, while the latter requires researchers to code and interpret the text in relation to the research question.
A representative study using generative probing examined whether models could reproduce American respondents’ descriptions of the two major US political parties. Drawing on respondents’ demographic profiles—including party affiliation, ethnicity, gender, income, and age—the researchers created a background statement written in the first person for each participant, then asked an LLM to describe the supporters of the two parties in four words. The model’s descriptions closely resembled those provided by human respondents.
Such similarity, however, does not establish that the simulation is entirely free of bias. For example, a model may avoid responses deemed offensive or “inappropriate.” Faced with moral conflicts—such as those between individual and community values, or personal integrity and professional ethics—it may also refuse to answer or respond evasively, introducing systematic bias into the resulting sample.
The validity of silicon samples can be assessed at three levels: how responses are elicited, the data produced, and the inferences drawn from it.
When eliciting responses, researchers must first test whether the model can accurately distinguish the concepts under study. If, for instance, the model treats theoretically distinct concepts such as “class” and “stratum” as synonymous, its responses may not reflect how the researcher intended those concepts to be measured. Subtle changes in prompt wording, the role information supplied, and the order of questions can also substantially alter the output. Incomplete demographic profiles may obscure important differences between groups, while repeatedly adjusting prompts until they yield the expected answers introduces selection bias.
In the data produced, LLMs may understate individual differences while amplifying stereotypes. Models trained to align with human values tend to give socially approved, cautious answers, resulting in less variation between simulated respondents than is observed in human samples. At the same time, group stereotypes embedded in internet training data may be incorporated into simulated roles, generating associations that do not exist in the target population. Content filtering and the reproduction of stereotypes may introduce different kinds of bias, but both can skew the distribution of simulated responses away from that of the target population.
Drawing inferences from silicon samples involves two stages: The LLM first generates responses using information in its pretraining data, and the researcher then uses those responses to draw conclusions about a real-world population. Measurement bias introduced during response generation may carry over into—or be amplified in—those conclusions. Although a model can generate a virtually unlimited number of responses, increasing the sample size can only reduce the effects of random variation. It cannot correct systematic bias arising from the model version, prompt design, or training data.

 

Social simulation driven by generative agents
Whereas research using silicon samples focuses on individual responses, agent-based modeling (ABM) examines how interactions among simulated individuals, or agents, give rise to broader social patterns. Each agent has specified attributes and follows particular behavioral rules. In traditional rule-driven ABM, researchers must define these attributes and rules, along with the conditions of the simulated environment, in advance. Simulating complex interactions involving language often requires those rules to be elaborated further.
Generative agent-based modeling (GABM) uses LLMs to guide agents’ decisions and interactions, allowing their actions to reflect their assigned roles and previous exchanges. Researchers can use role descriptions to represent individual differences and memory modules to retain a record of past interactions, while prompts and external programs jointly set limits on the agents’ actions.
Researchers have already used these methods to study how information spreads through social networks and to simulate interactions on a large scale. S3 (Social Network Simulation System) incorporates LLM agents into a simulated social network, using real-world data to model and assess how information, attitudes, and emotions spread. AgentSociety has simulated more than 10,000 agents and millions of interactions, producing results consistent with existing real-world research in several experimental scenarios. YuLan-OneSim allows researchers to construct scenarios using natural language and run simulations involving vast numbers of agents.
These examples demonstrate potential applications of GABM, but drawing conclusions about the real world requires examining how the conditions under which simulations run affect their results. First, model outputs are sensitive to prompts, sampling parameters, and model versions. A provider’s update to a model or its safety policy may cause the same experiment to yield different results, complicating comparisons over time and replication. Limited transparency about pretraining data and model updates also makes it difficult to fully explain how simulation results were produced. Second, training and alignment data reflect particular languages and cultures. If English and Western contexts predominate in that data, models may produce distorted generalizations about the roles, norms, and behavioral motivations of non-Western societies. Stereotypes embedded in the training data may also be repackaged as agents’ “individual differences.” Researchers should therefore assess whether GABM findings apply beyond the simulation by testing them against data from real people in the target setting, and clearly specify the populations and contexts to which their conclusions apply.
 
Debates, limitations of LLM-based simulation
Scholars are sharply divided over whether LLMs can serve as a reliable source of data. The debate centers on two questions: Whether model outputs can match human responses on particular tasks, and whether such a match indicates that models think and act through the same underlying mechanisms as humans.
Studies have found that, in behavioral economics experiments, social surveys, and political discourse tasks, LLMs can generate individual responses—or overall patterns of responses—that resemble those of humans. Some experiments presenting hypothetical scenarios in psychology and management research have also yielded results close to those obtained from human samples.
Other studies, however, have found that similarities observed within a particular task do not necessarily hold across tasks, models, and languages. Even when a model reproduces human responses on certain tasks, similar outward behavior does not establish that the underlying mechanisms are the same. LLMs predict what text comes next primarily from their training data and the current context. They lack the embodied human experience of warmth, cold, or pain, and do not directly experience the consequences of resource scarcity, competition for survival, or social pressure. A model can generate arguments for installing an elevator, for example, but explaining whether a household will actually help pay for it requires considering practical constraints, including its budget and how much it stands to benefit.
Humans and models also differ in the knowledge they draw on to act. Human decision-making relies heavily on “tacit knowledge” developed through physical activity, professional experience, and social interaction, including practical judgment and habitual ways of doing things. Such knowledge may never be recorded in the texts used to train models. Even when a model can restate the relevant rules, it may be unable to make the same trade-offs as a real person in an unfamiliar situation.
The effects of value alignment on social simulation must likewise be assessed in relation to what researchers intend to measure. Safety training and reinforcement learning from human feedback may lead models to favor socially approved responses emphasizing cooperation and altruism. The feedback data and how it is combined may also narrow the range of values expressed, making model responses more uniform on certain political or moral rating scales. Researchers must therefore distinguish these socially approved answers from how people in the target population actually behave.
LLMs used in social simulation should therefore be treated as measurement tools requiring calibration against empirical data, rather than as complete substitutes for human participants. Their widespread use also introduces a potentially reflexive relationship: The models used to simulate society may themselves influence it through interactions with people, reinforcing existing ideas or shaping changes in social attitudes. Further work is needed in both research practice and institutional standards to strengthen empirical calibration, test the mechanisms underlying simulated behavior, and refine methodological guidelines. Researchers must also clarify what simulation results can reasonably explain and under what conditions they can be applied.
In research practice, models should be calibrated against survey or behavioral data from people in the target population. Independent human samples should also be reserved for checking the distribution of simulated responses, the size of observed effects, and differences between groups. Methods reports should document the model’s name and version, system prompts, role settings, sampling parameters, how refusals were handled, and the number of repeated model calls. To identify bias and assess the robustness of findings, researchers should compare responses across different models, prompts, and languages, and repeat experiments at different times. For GABM, researchers should integrate LLMs with behavioral theory, causal structures, or explicit rules. They should separately assess how role settings, memory, environment, and interaction modules affect broader social outcomes, then test and refine the simulations against real-world data.
At the institutional level, standards should be established for reporting and evaluating methods used in LLM-based social simulation. This requires clear documentation of agent settings, interaction protocols, model updates, evaluation metrics, and human intervention. Baseline LLMs that have not undergone value alignment could provide a reference point for estimating bias, subject to strict access controls and ethical review. It is also necessary to support open-source simulation frameworks that can be deployed locally and readily adapted to local cultural contexts, enhance the verifiability and reproducibility of research, and improve relevant ethical guidelines.
 
Chen Huashan (associate research fellow) and Qu Chenji (assistant research fellow) are from the Center for Social Climate Research at the Chinese Academy of Social Sciences.

Edited by WANG YOURAN