NTCIR-5 CLIR Task

last update: Oct./11/2005

[NTCIR Home] [NTCIR Workshop 5 Meeting Home]

What's New

Oct./11/2005 No. of topics for each document sets
Total numbers of topics for evaluation are as follows.
Type of run No. of topics Removed topics
X->C 50 none
X->J 47 021,023,039
X->K 50 none
X->E 49 026
X->CJKE 50 none
note: X is C, J, K or E.

Reminder: deadline of submitting working notes
(1) For groups not using English docs: Oct./15/2005
(2) For groups using English docs: Nov./01/2005
Please visit to http://research.nii.ac.jp/ntcir-ws5/

Oct./02/2005 Release of trec_eval results for search runs using English document sets (E and CJKE)
Evaluation results for search runs using English document sets (E and CJKE) becomes available. Please download it.
CAUTION: to participants using English document sets.
We remove three topics (No.026) from our evaluation by trec_eval due to the fact that those topics have almost no relevant document. Therefore, in the case of using English document sets, the total number of is only 49.

Sep./19/2005 Ave/Med/MAX/Mis statistics
Statistics on average, median, max and min of mean average precision (MAP) by search type (eg., C-C-T) become available. Please download csv files (condensed into a file CSV.zip) from your web page for downloading evaluation results (Please access eval.html by the same ID and password as before. see below).

Sep./13/2005 Release of evaluation results
Delivery of evaluation results begins (except those for runs using English document sets).
Please download it from our server. Its URL is
http://rcir.nii.ac.jp/ntcir/access/XXX/YYY/ntcir5/clir/eval.html
Note:
(1) XXX and YYY have been already assigned to each participant respectively (same as those in document and topic release).
(2) ID and password for access are also the same as those for download of document and topic.
(3) Please access to "eval.html" file, not "index.html"file!
CAUTION: to participants using Japanese document sets.
We remove three topics (No.021, 023 and 039) from our evaluation by trec_eval due to the fact that those topics have almost no relevant document. Therefore, in the case of using Japanese document sets, the total number of is only 47. (JAPANESE ONLY)

Sep.13/2005 Information for writing your working note

Please visit to http://research.nii.ac.jp/ntcir-ws5/


Old Announcements

Sep./08/2005
*Results of evaluating runs on CJK documents will be delivered to participants on Sep./15.
*Deadline of submitting working notes is Oct./15
(Except the groups
submitting runs of English documents)

Aug/30/2005
Delay of delivery of evaluation results
We are so sorry but delivery of evaluation results is prolonged for some reasons. The date will be announced later. The deadline of submitting working notes is also changed.


NOTE: Deadline of submitting search runs using English Document is August/15/2005 23:59 (Japanese time)
*Additional submission of runs using CJK docs is also accepted by the same time.

July/07/2005 Release of English document collection for evaluation
If you have a plan to submit rersults searching English document collection in any sub-task, Please let us know. We will send information for downloading Xinhua dataset to you. You can obtain more infomation about English document sets for evaluation from the list of document sets.
*The number of documents in Daily Yomiuri 2000 was revised at 19:00 on July 07 (Japanese Time).

May/18/2005 Submission Guideline Release
The guideline for submitting search results is now available. Please read it carefully.

May/01/2005 Topic Release
The participants can download topic files from the NII server (URL and password have already been assigned at the time of document release). Please see 5.2 Topics in CFP about the format, and read cafully 6.Types of Runs in CFP before executing your search runs.

CAUTION: Please use 2000-01 years' doc files for evaluation. The 1998-99 years' files are just for training. DO NOT include any records of 1998-99 files into your search results that you will submit to us.

Jan./20/2005 Error Notice - CIRB040
  • Unfortunately, we find that the udn2000 file includes 3 error records, in which the <DOCNO> field contains some multi-byte codes.The udn2000 file is in the CIRB040 dataset, which is a NEW Chinese document set.
  • If you have already downloaded the CIRB040, you need to
    (1) download udn2000r file and replace the old udn2000, or
    (2) download CIRB040r and use it.
  • If you have not yet downloaded,
    please download CIRB040r (revised).
    (The udn2000r and CIRB040r are modified datasets.)
  • The method for downloading datasets is the same as before.
Nov./12/2004 Document release will begin on Nov./20/2004

Aug./20/2004 Release of Call for Participation

Call for Participation Note: Deadline of application is Sep./30/2004

Schedule

2004/09/30 Deadline of Registrations for Participation
2004/11/20 Release of Data (Document sets)
2005/05/01 Distribution of Search Topics
2005/06/01 Submission of Search Results
2005/09/13* Delivery of Evaluation Results
2005/10/15** Deadline for Working Notes
2005/12/06-09 NTCIR Workshop 5 (Conference)

Note:
* Only results of search runs using CJK document sets.
** Except groups submitting runs on English document sets.


If you have any questions, please contact to
@nii.ac.jp
All right reserved: National Institute of Informatics (NII) Japan