TREC 2026
Guidelines for Participants
Draft version, 2026-07-13
Track Overview
User simulation provides a scalable, cost-effective alternative to expensive human user studies when evaluating interactive search systems. However, the community currently lacks standardized methods for validating these simulators. The TREC User Simulation Track aims to establish a systematic framework for validating user simulators, determining the criteria for a reliable simulation, and creating best practices for simulation-based evaluation. For a comprehensive look at our long-term vision, please review the track’s Research Agenda.
The first edition of the track lays the groundwork for evaluating conversational simulators, with the specific goals of delivering (1) a robust standardized infrastructure for simulator-agent interactions, (2) a foundational dataset of both human and simulated conversational search logs, (3) initial evaluation baselines and behavioral metrics, and (4) a learned model capable of automatically predicting human-likeness scores.
Tasks
In this track, participants will develop a user simulator that mimics the behavior of a researcher using a conversational assistant to find datasets to help answer a specific research question.
Crucially, the conversational search assistant (the “agent”) will be provided by the track organizers. The sole focus for participating teams is to generate the user side of the dialogue, accurately simulating how a human researcher would query, clarify, and respond to the agent.
Participants can evaluate their simulators through two distinct tasks:
Task 1: Turn-level Next Utterance Prediction
This task focuses on a simulator’s ability to model the immediate, reactive behavior of a user at a single turn. It tests local conversational coherence and behavioral realism within an ongoing dialogue.
- Input: A scenario and a partial conversation history (a sequence of preceding user and system turns) and the simulated user’s underlying initial information need.
- Output: The single, predicted next user utterance. Participants may also include associated dialogue acts representing the semantic intent of the simulated utterance.
Task 2: Session-level End-to-End Conversation Generation
This task evaluates a simulator’s ability to strategically manage an entire conversation to achieve a predefined goal. It tests high-level planning, conversational persistence, and the simulator’s ability to recognize task success.
- Input: A scenario that specifies the simulated user’s persona and goals.
- Output: A complete, multi-turn conversation. The simulator must dynamically interact with the provided system, generating sequential turns until the simulator autonomously decides the goal is satisfied or that the search should be abandoned.
Scenarios
In traditional TREC tracks, systems are evaluated against a standardized set of “topics.” In the User Simulation Track, these are defined as scenarios. A scenario provides the foundational context that drives the simulated user’s behavior. Each scenario is constructed from three core components:
- Persona: The defining characteristics of the simulated user. This encompasses their background, communication style, and general technical proficiency.
- Goal: The specific information need. This details the exact type of dataset the user is attempting to find, as well as the broader research context or problem that is motivating their search.
- Persona-goal interaction: The intersection of the user’s background and their specific objective. This component defines the user’s domain expertise and familiarity with the subject matter.
Simulators must synthesize all three components of the scenario to maintain a consistent, realistic behavior throughout the conversation.
Training and Test Data
- Training data (participant-sourced): During the topic development phase, participating teams will be asked to interact directly with our development agents to accomplish a set of predefined training goals. For this collection, participants will act as themselves—reflecting their genuine characteristics, technical proficiency, and search habits rather than adopting a fictitious persona. The resulting interaction logs will be distributed to all teams to serve as the primary training and validation dataset.
- Test data (held-out evaluation): To establish a robust human baseline for the final evaluation, the organizers will collect a separate set of human conversations. This collection relies on a strictly distinct set of test goals and will be sourced independently (e.g., via organizers, students, and crowdsourced workers). The conversational logs from this test set will remain private during the submission phase and will be used to benchmark the submitted official runs.
In addition to the core datasets above, we will release an external dataset focused on the exact same task (conversational dataset search). Crucially, this dataset includes think-aloud transcripts. While these logs were collected using different conversational agents than the ones provided in this track, the think-aloud data is specifically meant to provide deep insights into how real humans cognitively approach and strategize completing the dataset search task. This supplemental resource is intended to help teams bootstrap the early design of their simulators before the official training data becomes available.
Setup (API and Infrastructure)
The evaluation environment relies on four components (illustrated in the figure above):
- Simulators: The user simulation models developed by participating teams.
- Conversational Agents: Systems designed for dataset search, provided by the track organizers, which the simulators will interact with.
- TREC UserSim API: The central broker that facilitates and standardizes all communication between users (simulated or human) and the conversational agents.
- Web Interface: A frontend exposing the conversational agents to human users, utilized by the organizers to collect baseline human interaction data.
Conversational Agents
The track provides two categories of conversational agents to assess how robustly a simulator adapts to different system behaviors and capabilities:
- Development Agents: Released during the initial phase to help participants design, tune, and test their simulators.
- Test Agents: Used exclusively during the data collection and official submission phases. These are the systems your simulators will interact with to generate the final submitted runs.
TREC UserSim API
The TREC UserSim API is the central gateway for all interactions in this track. It standardizes the communication between user simulators and the provided conversational agents, handling both the development phase and the orchestration of official run submissions. Comprehensive documentation covering API endpoints, authentication, and expected request/response JSON formats is provided in the track’s GitHub repository (link coming soon).
Submission Guidelines
Similar to TREC iKAT, the User Simulation Track relies on interactive submissions. Instead of submitting static document rankings, participants connect their user simulators to a live API endpoint (think of it as submitting user simulators instead of system rankings). Concretely, a run submission comprises the utterances of a single user simulator across multiple conversations, constructed from combinations of scenarios and conversational agents.
Participants may submit up to 4 official runs per task that are included in the evaluation. In addition, they are granted (n+1)×10 debug runs, where n is the number of training conversations contributed by the participating team, to support the development of their user simulators prior to official submission. Both official and debug runs use the same API, but with distinct endpoints. API access tokens will be distributed upon registration.
The user simulator completes a run submission, covering multiple scenario-agent combinations. Each combination results in a conversation with multiple turns between the user simulator and the conversational agent. The workflow for completing the run is a stream of client-side requests (from the user simulator) and responses from the infrastructure (based on the conversational agent).
Task 1: Turn-level Next Utterance Prediction
The client initiates the run with POST task1/run/start, which will return the first scenario and the chat history in the response. The client-side user simulator then generates the next utterance which is sent to the infrastructure and conversational agent with POST task1/run/continue. The response from the TREC UserSim platform contains the next scenario and the corresponding chat history for which the next utterance has to be simulated. The client repeats POST task1/run/continue requests and simulates next utterances until the run is completed.
POST task1/run/start
Response from the TREC UserSim platform:
{
"conversation_id": "5969273be4de4b21bbbe4e123f030f08",
"scenario": {
"goal": {
"context": "We are exploring ideas from philosophy of language, and linguistics, that describe how conversations are structured. …",
"topic": "Recordings of natural-language conversations either between two people, or a person and a machine, where the conversation is sustained (at least a few turns each), …",
"discipline": "Social Sciences"
},
"persona": {
"general_info": {
"gender": "Female",
"age": "18-34",
"highest_education": "PhD/Doctorate",
"proficiency_in_english": "Advanced",
"tools_used_for_dataset_search": [
"Hugging Face Datasets",
"Kaggle"
]
},
"experience_with_ai": {
"trust": "Very critical",
"perceived_human_likeness": "High anthropomorphism"
},
"individual_traits": {
"frustration_threshold": "Moderately quickly",
"interaction_style": "Elaborate"
}
},
"persona_goal_interaction": {
"domain_familiarity": "Moderately familiar",
"known_datasets": [
"MultiWOZ",
"TREC CAsT and TREC iKAT"
]
}
},
"chat_messages": [
{
"timestamp": "2026-06-17T05:06:57.999383",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "user",
"text": "I am looking for datasets of multi turn conversations. ...",
"sources": [],
"annotations": {},
"is_final": false
},
{
"timestamp": "2026-06-17T05:07:36.943677",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "agent",
"text": "Below are a handful of public resources that meet most (or all) of the criteria you mentioned ...",
"sources": [],
"annotations": {
"helpful": false,
"dataset_quality": "Unlikely to be useful",
"feedback": "The recommended datasets do not include multi-turn, goal-driven dialogue collections."
},
"is_final": false
},
{
"timestamp": "2026-06-17T05:18:57.504701",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "user",
"text": "Hmm, can you suggest any English, multi-turn, goal-driven dialogue collections?",
"sources": [],
"annotations": {},
"is_final": false
},
{
"timestamp": "2026-06-17T05:19:53.580027",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "agent",
"text": "Below is a list of English, multi-turn, goal-driven dialogue collections ...",
"sources": [],
"annotations": {
"helpful": false,
"dataset_quality": "Some promising datasets",
"feedback": ""
},
"is_final": false
}
]
}POST task1/run/continue
Payload of the request:
{
"run_id": "5d41402abc4b2a76b9719d911017c592",
"user_utterance": {
"timestamp": "2026-06-17T05:19:57.480027",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "user",
"text": "Can you provide more details about the second dataset you mentioned, such as its size, the number of conversations, and the types of goals it covers?",
"sources": [],
"annotations": {},
"is_final": false
}
}Task 2: Session-level End-to-End Conversation Generation
The client initiates the run with POST task2/run/start, which will return the first scenario in the response. The client-side user simulator then generates the first utterance which is sent to the infrastructure and conversational agent with POST task2/run/continue. The corresponding response contains the chat history, including the utterance made by the agent in response to the simulated user’s utterance. The user simulator continues the conversation by sending the next utterance with POST task2/run/continue until the conversation is finished.
POST task2/run/start
Response from the TREC UserSim platform:
{
"conversation_id": "5969273be4de4b21bbbe4e123f030f08",
"scenario": {
"goal": {
"context": "We are exploring ideas from philosophy of language, and linguistics, that describe how conversations are structured. …",
"topic": "Recordings of natural-language conversations either between two people, or a person and a machine, where the conversation is sustained (at least a few turns each), …",
"discipline": "Social Sciences"
},
"persona": {
"general_info": {
"gender": "Female",
"age": "18-34",
"highest_education": "PhD/Doctorate",
"proficiency_in_english": "Advanced",
"tools_used_for_dataset_search": [
"Hugging Face Datasets",
"Kaggle"
]
},
"experience_with_ai": {
"trust": "Very critical",
"perceived_human_likeness": "High anthropomorphism"
},
"individual_traits": {
"frustration_threshold": "Moderately quickly",
"interaction_style": "Elaborate"
}
},
"persona_goal_interaction": {
"domain_familiarity": "Moderately familiar",
"known_datasets": [
"MultiWOZ",
"TREC CAsT and TREC iKAT"
]
}
}
}POST task2/run/continue
Payload of the request:
{
"run_id": "5d41402abc4b2a76b9719d911017c592",
"user_utterance": {
"timestamp": "2026-06-17T05:06:57.999383",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "user",
"text": "Hello, I need datasets containing conversations to analyze how conversations are structured ...",
"sources": [],
"annotations": {},
"is_final": false
}
}Response:
{
"conversation_id": "5969273be4de4b21bbbe4e123f030f08",
"scenario": {...},
"chat_messages": [
{
"timestamp": "2026-06-17T05:06:57.999383",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "user",
"text": "Hello, I need datasets containing conversations to analyze how conversations are structured ...",
"sources": [],
"annotations": {},
"is_final": false
},
{
"timestamp": "2026-06-17T05:07:36.943677",
"conversation_id": "fb0686887ecf4d24b69ff5454e0ca1a8",
"participant_name": "agent",
"text": "Greetings, below you can find several datasets that meet your requirements ...",
"sources": [
"https://aclanthology.org/D18-1547/",
"https://trec.nist.gov/data/cast.html"
],
"annotations": {},
"is_final": false
}
]
}The conversation ends either when (1) the conversation budget is depleted (tracked by the infrastructure) or (2) the user simulator decides to end the conversation. Closing a conversation is handled by the boolean parameter is_final in the response or request body, depending on which party decides to end the conversation. In both cases, the infrastructure returns a response with status code 201 and information about the next scenario (if the budget is depleted both the status code and the new scenario are included in the same response). The user simulator then continues completing the next conversation for a new scenario. Technically, the user simulator sends POST /run/continue requests until the run is completed, i.e., all conversations are simulated for all scenario-agent combinations. Once the run is completed, it is signalled by the status code 428 to the user simulator.
To help participants get started, the organizers will provide a baseline simulator template that handles interaction with the infrastructure and conversational agents. Likewise, the organizers provide additional in-depth technical documentation of the entire REST-API in a dedicated repository, for instance, sequence diagrams for both Task 1 and Task 2.
Evaluation Measures
We will evaluate submissions using:
- Qualitative evaluation (human raters): A form of Turing Test, where trained annotators are presented with two dialogues and must attempt to guess which one is simulated.
- Behavioral metrics (quantitative log comparison): A comparison of simulated dialogues against real human usage logs, in terms of (the distributions of) key interaction behaviors (e.g., query length, turn count, clarification requests) and utilizing dialogue act annotations to model and compare semantic intent.
Timeline
- April: Draft guidelines released
- June: Training data collection begins (guidelines finalized)
- July: Training data collection complete
- July: System development begins (track API opens)
- September: Submissions due
- November: TREC Conference