TREC Conversational Assistance Track (CAsT)

Conversational search benchmark at TREC

TREC Conversational Assistance Track (CAsT)

There are currently few datasets appropriate for training and evaluating models for Conversational Information Seeking (CIS). The main aim of TREC CAsT is to advance research on conversational search systems. The goal of the track is to create a reusable benchmark for open-domain information centric conversational dialogues.

The track will run in 2022 and establish a concrete and standard collection of data with information needs to make systems directly comparable.

This is the fourth year of TREC CAsT, which will run as a track in TREC. This year we aim to focus on candidate information ranking in context:

This year, there’s an optional mixed-Initiative sub-task that evaluates the ability of systems to use mixed-initiative for more effective conversations. For more information, see the guidelines linked below.

Year 4 (TREC 2022)

Important Dates

Data

Topics

Mixed Initiative Question Pool

Corpora

Participants have the option of processing the collection (to generate passage splits) themselves using the provided tools or requesting the processed corpus from the organizers. You can make the request to the organization team via Slack or Google Groups (see below).

If you processed the corpora yourself, please verfify that you have the right passage splits by comparing the hashes of each passage with the master version.

Baselines

Duplicate handling

Guidelines

Run Validations

Contact

Organizers

Year 3 (TREC 2021)

Important Dates

Data

Topics

Corpora

Baselines

Duplicate handling

Guidelines

Contact

Organizers

Year 2 (TREC 2020)

News

Data

Topics

Baselines

Collection

Year 1 (TREC 2019)

2019 Data

Topics

Resolved Topic Annotations

Baselines

Collection

Document ID format

Duplicate handling

Code and tools