The Music Information Retrieval / Music Digital Library Evaluation Project J. Stephen Downie Assistant Professor University of Illinois at Urbana-Champaign This research project is designed to enhance the significant work being done by the Music Information Retrieval (MIR) and Music Digital Library (MDL) research communities by providing an opportunity for these communities to realize the establishment of sorely-needed evaluation tools. We are building upon the ongoing efforts being made to establish TREC-like and other comprehensive evaluation paradigms within the MIR/MDL research communities, and our research tasks are based upon expert opinion garnered from members of the Information Retrieval (IR), MDL and MIR communities with regard to the construction and implementation of scientifically valid evaluation frameworks. The two complementary tracks of our work are: 1) the establishment of internationally accessible mechanisms and evaluation standards for the comprehensive evaluation of MIR and MDL systems and 2) the formal investigation of the human factors involved in the creation, use and evaluation of MIR and MDL systems. We hope to achieve: 1. The creation and refinement of secure access mechanisms that will allow the manipulation of a unique, large-scale standard corpus of music materials for the research and evaluation use of the international MIR/MDL research community. 2. The creation, refinement and dissemination of a TREC-like evaluation scenario based upon the special needs and requirements of MIR/MDL community. 3. The creation, refinement and dissemination of a collection of standardized query documents, based upon the real-world expression of user needs. 4. The creation, refinement and dissemination of a deeper and more comprehensive understanding of the behavior of MIR/MDL systems, their uses and their users. http://music-ir.org/evaluation