INEX 2004INitiative for the Evaluation of XML Retrieval Call for participationThe DELOS Network of Excellence for Digital Libraries invites participation in an evaluation initiative for XML document retrieval. The widespread use of the extensible Markup Language (XML), especially the increasing use of XML in scientific data repositories, Digital Libraries and on the Web, brought about an explosion in the development of XML tools, including systems to store and access XML content. The aim of such retrieval systems is to exploit the logical structure of documents, which is explicitly represented by the XML markup, and retrieve document components, instead of whole documents, in response to a user query. Implementing this, more focused, retrieval paradigm means that an XML retrieval system needs not only to find relevant information in the XML documents, but also determine the appropriate level of granularity to return to the user. In addition, the relevance of a retrieved component is dependent on meeting both content and structural conditions. Evaluating the effectiveness of XML retrieval systems, hence, requires a test collection where the relevance assessments are provided according to a relevance criterion, which takes into account the imposed structural aspects. A test collection as such has been built as a result of two rounds of the Initiative for the Evaluation of XML Retrieval (INEX 2002 and INEX 2003). This initiative provides an opportunity for participants to evaluate their XML retrieval methods using uniform scoring procedures and a forum for participating organisations to compare their results. As part of a large-scale effort to improve the efficiency of research in information retrieval and digital libraries, this project initiated an international, coordinated effort to promote evaluation procedures for content-oriented XML retrieval. In INEX 2004, participating organisations will be able to compare the retrieval effectiveness of their XML document retrieval systems and will contribute to the continuous construction of a large XML test collection. The test collection will also provide participants a means for future comparative and quantitative experiments. Due to copyright issues, only participating organisations will have access to the constructed test collection. INEX test collectionThe test collection consists of a set of XML documents, topics and relevance assessments. The topics and the relevance judgments are obtained through a collaborative effort from the participants. Detailed guidelines on the on-line topic submission, retrieval result submission, relevance assessment task, and evaluation metrics will be provided by INEX. DocumentsThe INEX document collection is so far made up of the full-texts, marked up in XML, of 12,107 articles of the IEEE Computer Society's publications from 12 magazines and 6 transactions, covering the period of 1995-2002, and totalling 494 megabytes in size. The collection has a suitably complex XML structure (192 different content models in DTD) and contains scientific articles of varying length. On average an article contains 1,532 XML nodes, where the average depth of a node is 6.9. TopicsEach participating group will be asked to create a set of candidate topics, which are representative of the range of real user needs over the XML collection. The queries may be content-only (CO) or content-and-structure (CAS) queries, and broad or narrow topic queries. CO queries are free text queries, like those used in TREC, for which the retrieval system should retrieve relevant XML elements of varying granularity, while CAS queries contain explicit structural constraints, such as containment conditions. From the pooled set of candidate topics INEX will select a final set of topics to form part of the INEX test collection TasksThe general task, to be performed with the data and the final set of topics, will be the ad-hoc retrieval of XML documents. Similarly to information retrieval, we regard ad-hoc retrieval as a simulation of how a library might be used, where a static set of documents is searched using a new set of queries (topics). The main differences are that, in INEX, the library consists of XML documents, the queries may contain both content and structural conditions and, in response to a query, arbitrary XML elements may be retrieved from the library. Participants will be able to submit up to a fixed number of runs, each containing the top 1500 retrieval results for each of the selected topics. INEX will have this year four tracks:
Relevance assessmentsRelevance assessments will be provided by the participating groups using INEX's on-line assessment system. Each assessor will judge 1-2 topics, either the topics that they originally created or if these were removed from the final set of topics, then topics that were similar to their original queries. Please note that assessments will take about one person week per topic. Participating groups will gain access to the completed INEX test collection only after they have completed their assessment task. EvaluationThe evaluation of the retrieval effectiveness of the XML retrieval engines used by the participants will be based on the constructed INEX test collection and uniform scoring techniques, including recall/precision measures, which will take into account the structural nature of XML documents, and overlap of answers. Participants will be able to present their approaches and final results at the INEX 2004 workshop to be held in December in Dagstul. All results will be published in the INEX workshop proceedings and on the Web. Data Handling AgreementIn order to have access to the data designated as the IEEE Computer Society XML Retrieval Research Collection, organizations (who did not sign the agreement in 2003) must first fill in a data release Application Form (to be obtained from the INEX 2004 web site). ScheduleApril 2: Deadline for the submission of "Application for Participation". April 02 - 16: The collection of XML documents will be distributed to all participants on the receipt of their signed data handling agreement. Participants will also be provided with detailed instructions and formatting criteria for candidate topics/queries. May 03: Submission deadline for candidate topics. May 24: Distribution of final set of topics/queries to participants along with detailed information on the formatting requirements of the search results. August 09: Submission deadline of search results. August 23: Distribution of merged results to participants for relevance assessments. October 08: Submission deadline for relevance assessments. Nov 01: Distribution of XML test collection and evaluation scores to participants. December 1 (tbc): Submission of papers for the workshop pre-proceedings December 13-15 (tbc): Workshop in Schloss Dagstuhl (http://www.dagstuhl.de/). OrganisersProject Leaders Mounia Lalmas Contact person Topic format specification Andrew Trotman Online relevance assessment tool Metrics Arjen P. de Vries |