Relevance Evaluation. For assessing the relevance, we used the test queries de- fined above. The queries were issued to top-k sources selected by different source selection methods. The top-5 results returned were manually classified as rel- evant or irrelevant. The classification was rule based. For example, if the test query is “Pirates Caribbean Chest” and the original movie name is “Pirates of Caribbean and Dead Man’s Chest” then if the result entity refers to the movie “Pirates of Caribbean and Dead Man’s Chest” (DVD, Blue-Ray etc.) then the 0.5 0.4 0.3 0.2 DSR(0.9) TSR(0.9) 0.6 DSR(0.9) TSR(0.9) 0.5 0.4 0.3
