AI Synthetic Users Match Human Conclusions but Miss the Nuance, Study Finds

Increasing numbers of research teams are adopting synthetic users—artificial intelligence simulations of study participants—to conduct investigations without the expense and logistics of recruiting actual people. Following earlier work examining synthetic users in interview settings, researchers recently evaluated their performance on a narrower task: providing answers to a single open-ended survey question.

The test reused a question from a prior investigation into UX terminology, where 604 technology and design professionals described what they would call UX work. The same question went to 120 synthetic respondents, and the resulting datasets were compared. While both pointed toward identical broad conclusions, the synthetic group's responses showed substantially less diversity than those from the human group.

How the Synthetic Respondents Were Generated

In February 2026, researchers obtained synthetic responses through Synthetic Users, a generative-AI platform designed to simulate research participants and their behavior.

The team created 120 synthetic respondents intended to represent technology and design professionals across the globe. They supplied Synthetic Users with the research plan, encompassing research questions, inclusion and exclusion criteria, and survey items. The platform then automatically built a profile for each respondent, assigning characteristics including job title, age, location, organization type, years of experience, cultural background, and Big Five personality traits.

Researchers asked the synthetic respondents to answer the identical open-ended question from the original human study:

Question: Which professional field or discipline do you believe best describes the following work? An individual gathers insights about people's needs and experiences and turns those insights into recommendations for how a product or service should be planned, built, and improved. Please name only 1 field or discipline.

Two platform constraints fell outside researcher control. First, Synthetic Users operates through simulated interviews, causing synthetic respondents to answer as though speaking with an interviewer and frequently producing longer responses than the human survey answers. Second, the platform occasionally rephrased the question—substituting "people's needs, behaviors, and experiences" for "people's needs and experiences," for instance. While such rewording may aim to replicate how human interviewers naturally rephrase questions in conversation, even minor wording shifts can influence how respondents interpret a question, making it impossible to rule out their effect on synthetic responses.

These constraints highlight a fundamental challenge with synthetic-user platforms: researchers possess limited authority over question presentation, and that presentation could function as a confounding variable in research.

Both datasets underwent identical analysis. Since certain respondents named multiple fields, researchers coded each response for keywords rather than forcing responses into single categories.

The Synthetic and Human Data Converged on the Same Overall Result

The two datasets reached the same fundamental finding: UX terminology has not been displaced as the prevailing designation for this category of work.

Eighty-four percent of synthetic respondents employed UX-related terminology, in contrast with roughly 70% of human respondents. This 14-percentage-point gap achieved statistical significance. However, this difference would not alter the practical takeaway from the original investigation: UX clearly dominated terminology in both groups.

Context matters for understanding this gap. The human sample came partly through the researchers' own networks, including their Practitioner Panel. As documented in the original study, this recruitment approach likely produced a sample with heightened familiarity with UX terminology, potentially inflating estimates of how widely "UX" circulates among technology and design professionals.

The synthetic respondents did not come through these channels and therefore lacked this particular sampling bias. The human dataset comprised 67% UX professionals, whereas only 20% of the synthetic dataset held UX roles. This disparity was intentional: researchers established quotas for different professional roles when constructing the synthetic sample to ensure broader professional representation, and the final synthetic sample closely aligned with these quotas.

In principle, the synthetic dataset presented a chance to represent a wider spectrum of technology and design professionals. Yet UX-related terminology appeared even more frequently in the synthetic data than in the already UX-focused human sample.

The exact cause remains unclear from this comparison. However, earlier studies of synthetic respondents have documented that their answers tend to cluster around typical or predictable responses, exhibiting less of the natural variation present in actual human populations.

Synthetic Respondents Used a Narrower Range of Terms: Mode Collapse

Human respondents deployed a substantially broader vocabulary than their synthetic counterparts. The human group generated over 50 distinct labels, encompassing terms like "consumer-behavior research," "digital-experience strategy," "journey management," and "human-factors engineering." Most of these alternatives surfaced only once or a small number of times.

Synthetic respondents, conversely, generated a far more limited set of responses collectively. The disparity became most striking in less-frequent answers: synthetic respondents produced none of the specialized or unusual terms appearing in the human data, such as "digital management" or "business analysis."

This pattern mirrors mode collapse—a machine-learning concept describing a generative model that repeatedly outputs a narrow band of responses rather than the complete spectrum it was trained to produce. The term draws from the statistical definition of "mode": the most frequently occurring value in a dataset.

In synthetic-users research, mode collapse refers to synthetic data that reproduces the most common data point but drops the long tail of less-common responses in the target human population.

The synthetic respondents concentrated on roughly a dozen labels; the human group dispersed across more than 50.

This narrower vocabulary stands out given the response format. Since synthetic respondents answered through an interview format, their responses extended longer than the humans' and offered additional chances to reference alternative or related terms. Yet they still produced fewer distinct labels.

Synthetic respondents omitted several common keywords, including "human factors," "experience design," or "consumer research." (The percentages are not mutually exclusive: a participant may have used several terms.)

Whether this gap matters hinges on research objectives. For studies seeking only the most prevalent response, synthetic data may suffice. For investigations aiming to surface less-frequent responses—surprising terminology, emerging patterns, or avenues for deeper inquiry—human participants remain essential.

Researchers might attempt to address mode collapse through prompting a model to generate more varied answers. Yet producing greater variety does not guarantee that variety reflects reality. The model would need to generate the kinds of uncommon responses that actual people provide, and at the frequencies those responses actually occur. Absent human data for reference, determining whether that variation authentically mirrors the population becomes difficult.

Job Role Mattered, but "UX" Was Still More Prevalent

Professional role shaped terminology in the synthetic dataset, just as it did in the human dataset. In the synthetic data, 45% of customer-insights professionals incorporated the term "customer" in their response, whereas none of the synthetic UX professionals did. Similarly, 30% of the synthetic market-research professionals used "market" in their response, compared with 0% of the synthetic UX professionals. These differences reached statistical significance.

Nevertheless, synthetic respondents outside UX still favored UX-related terminology more than their human equivalents. For example, 93% of product managers in the synthetic dataset deployed UX-related terminology, versus 44% of product managers in the human dataset.

The synthetic data did not eliminate professional background's influence, but it pushed respondents across different roles toward the dominant label more forcefully than the human data did. Mode collapse operated both across the entire sample and within each professional subgroup.

This carries weight for researchers hoping to use synthetic respondents to examine distinctions between audiences. Synthetic data may still demonstrate that groups think or communicate differently, yet it may understate those distinctions by drawing responses toward the most common framing. In open-ended investigations of how different groups naturally describe a concept, that erosion of variation could obscure differences that human participants would uncover.

Synthetic-User Research Is Still Not User Research

The synthetic respondents replicated the core finding of the original study—that UX persists as the field's dominant term—but they did not replicate the complete range of variation in how real people responded. They surfaced fewer uncommon alternatives and demonstrated a stronger gravitational pull toward UX terminology across professional roles: mode collapse at both the aggregate level and within each subgroup.

Research involving synthetic users remains distinct from user research, since no actual users participate in the investigation. Treat synthetic findings as preliminary rather than as confirmation of what users actually think, say, or do. A plausible finding—even one that aligns with human research—does not prove that synthetic respondents will faithfully represent the less-common viewpoints, subgroup distinctions, or organic variation that may prove crucial to your research question. Deploy synthetic respondents to develop hypotheses or validate broad trends; when the less-common responses carry significance, human participants are necessary.