Background
Machine Reading Comprehension (MRC) enables computers to read, process, and understand natural language text, considered to be one of the core abilities of artificial intelligence. It is of great value for next-generation search engines and intelligent agent products, and has received wide attention across academia and industry in recent years.
With the intention of advancing reading comprehension technology, 2018 NLP Challenge on Machine Reading Comprehension is organized jointly by Chinese Information Processing Society of China(CIPS), China Computer Federation(CCF), and Baidu Inc.. The challenge provides a large-scale, open-domain, application-oriented Chinese MRC dataset and a platform for research and academic exchanges on MRC, NLU and other AI technologies and applications. The workshop and award ceremony of the challenge will be held at the third Language & Intelligence Summit. All researchers and developers are welcomed.
About the Challenge
1. Task Description
Given a question "q", and a set of documentsD = d1, d2, ..., dn，the participating MRC system is expected to output an answer "a" that best answers "q" based to the evidences in D.
2. Dataset
The dataset contains 300k questions sampled from real anonymized user queries from Baidu Search. Each question has 5 corresponding evidence documents and human generated answers. The dataset is divided into a training set (270k questions), a development set (10k questions) and a test set (20k questions). A subset of 200k questions have already been released in DuReader dataset, available for free download and using for pre-training/validation. Competition participants will get the new data of 100k questions after the registration deadline.
3. Evaluation Metrics
ROUGH-L and BLEU4 are adopted as the basic evaluation metrics to measure the performance of participating systems, with the former as the main measurement. Some minor modifications are made over the original ROUGE-L and BLEU4 metrics to better measure the performance of YES-NO and ENTITY type questions.
*Please refer to the specification enclosed in the dataset package for details.
4. Baseline Systems
Participation Info
1. Eligibility
The challenge is open to all individuals, research institutions, colleges, universities, and enterprises in related field.
2. Registration
The challenge opens to registration on March 1st, 2018.
Please go to the official website to register.*Teams who registered and submitted valid results will get a Memorial T-shirt for each member.
3. Registration Deadline
March 31st, 2018
Scan QR code to join in MRC Challenge group
Timeline
Mar 1
Registration open, partial data release
Mar 31
Registration close, full data release
Apr 23
Test data available
Apr 30
Testing results submission due
May 15
Final results announcement, system report submission
Jul 28
Workshop and award ceremony
Awards Setting
The challenge will award one First Prize, two Second Prizes and three Third Prizes. Winners will get the award certificates issued by CIPS & CCF. The prizes and travel grants for attending the workshop and award ceremony will be sponsored by Baidu Inc..
¥50,000
award certification
¥20,000
award certification
¥3,000
award certification
*Notes:
1. All prizes are inclusive of taxes.
2.The award requires participants to provide their system reports (including method descriptions, system code & data, references, etc.) and name lists of team members.
Organization
1. Hosts
Chinese Information Processing Society of China (CIPS)China Computer Federation (CCF)
2. Organizer
Baidu Inc.Committee on Evaluation of CIPS (CIPS CE)Technical Committee on Chinese Information Technology of CCF (CCF TCCI)
3. Steering committee
Le SunInstitute of Software, Chinese Academy of SciencesMing ZhouMicrosoft Research Asia
Erhong YangBeijing Language and Culture UniversityDongyan ZhaoPeking University
Hua WuBaidu Inc.
4. Organizing committee
Yajuan LyuBaidu Inc.Xianpei HanInstitute of Software, Chinese Academy of Sciences
Xiaojun WangPeking UniversityKai LiuBaidu Inc.