Xxxxxxxxx and Risks. A common cry from academic evaluation researchers is a request for logs, but many years of asking have provided few widely available datasets of user interactions. We have to be more creative than calling for others to help us. A key risk is that the smaller scale research and results that we conduct will not translate to the large scale problems of the search engines. However the only way to understand such risks will be to try.
Xxxxxxxxx and Risks. While some of the proposed research activities (metrics, performance analysis, assumptions) can already be carried out with existing resources, the identification of performance-critical applica- tion features and the development of performance models require empirical data from a larger variety of test collections. Thus, researchers should share their test collections both for support- ing reproducibility and research on prediction. Indeed, while individual contributions to such an effort might not seem worthwhile for researchers, collaborative approaches in the form of eval- uation campaigns might be more promising. Another potential obstacle is the need for more sophisticated competencies in data modelling, statistics, and data analysis, and so on. Moreover, both the explanatory and the predictive performance models may be quite demanding in terms of computational resource needed to train and compute them.
Xxxxxxxxx and Risks. The problem of evaluation of interactive search support is extremely difficult to solve, if comparison and generalization of results is to take place. There does not currently exist a sound, generally accepted theoretical understanding or model of interactive IR, on the basis of which the evaluation criteria, measures and methods can be derived.
