Agreement Based Measures. We describe the two baselines derived based on agree- ment based methods, which are directly derived from SourceRank [10]. We compare with these measures as standalone and in combination with query-relevance based measures. Undifferentiated SourceRank, USR: USR does not differentiate between topics. We created a single agreement graph across the topics based on entire set of sampling queries (Section 4.1). The USR of sources are computed based on a random walk on this agree- ment graph. Combining USR with query similarity based source quality measures like XXXX is an intu- itive way of extending USR to multiple topics. Note that TSR for a topic was computed based only on the sampling queries in that topic. In other words, TSR predominantly considers endorsement of sources from the topic, where as USR considers endorsement from every sources equally. Comparing USR and TSR performances is an excellent validation of whether the topic specific endorsement improves result quality.
