Common use of OVERALL PERFORMANCE Clause in Contracts

OVERALL PERFORMANCE. To answer our first research question, whether embeddings improve the score of entity retrieval, we compare our entity re-ranking approach with a number of baseline entity retrieval models. Table 2 shows the results for different models with respect to NDCG@10 and NDCG@100, the default evaluation measures for DBpedia-entity V2. In this table, the embedding-based similarity component (Eq. (5)) is denoted by ESim, where c and cg subscripts refer to the two versions of our entity embeddings: without and with link graph. Table 2. Results of embedding-based entity re-ranking approach on different query subsets of DBpedia-Entity V2 collection. Significance of results is explained in running text. Model SemSearch INEX-LD ListSearch QALD-2 Total NDCG @10 @100 @10 @100 @10 @100 @10 @100 @10 @100 Reranking the FSDM top 1000 entities ESimc 0.365 0.412 0.194 0.252 0.210 0.288 0.192 0.255 0.239 0.300 ESimcg 0.397 0.462 0.216 0.282 0.211 0.311 0.213 0.286 0.258 0.334 FSDM 0.652 0.722 0.421 0.504 0.420 0.495 0.340 0.436 0.452 0.534 +ELR 0.656 0.726 0.435 0.513 0.422 0.496 0.347 0.446 0.459 0.541 +ESimc 0.659 0.725 0.433 0.513 0.432 0.509 0.353 0.447 0.463 0.543 +ESimcg 0.672 0.733 0.440 0.528 0.424 0.507 0.349 0.451 0.465 0.549 Reranking the BM25F-CA top 1000 entities ESimc 0.381 0.424 0.194 0.253 0.211 0.283 0.192 0.252 0.243 0.301 ESimcg 0.417 0.478 0.217 0.286 0.211 0.302 0.212 0.282 0.262 0.335 BM25F-CA 0.628 0.720 0.439 0.530 0.425 0.511 0.369 0.461 0.461 0.551 +ESimc 0.658 0.730 0.462 0.545 0.448 0.529 0.380 0.469 0.481 0.563 +ESimcg 0.660 0.736 0.466 0.552 0.452 0.535 0.390 0.483 0.487 0.572 ± ± The results of our method are presented for components ESimc and ESimcg by themselves (i.e., λ =1 in Eq. (6)), and also in combination with FSDM and BM25F-CA. The mean and standard deviation of λ found by the Coordinate Ascent algorithm over all folds are: 0.34 0.02 for FSDM+ESimc, 0.61 0.01 for ± ± FSDM+ESimcg, 0.81 0.03 for BM25F-CA+ESimc, and 0.88 0.00 for BM25F- CA+ESimcg. The results show that the embedding-based scores alone do not perform very well, however, when combining them with other scores, the per- formance improves by a large margin. We determine the statistical significance of the difference in effectiveness for both the NDCG@10 and the NDCG@100 values, using the two-tailed paired t-test with α < 0.05. The results show that both versions of FSDM+ESim and BM25-CA+ESim models yield significant improvements over FSDM and BM25-CA models (with respect to all metrics), respectively. Also, FSDM+ESimcg improves significantly over FSDM+ELR with respect to NDCG@100, showing that our embedding based method captures entity similarities better than the strong entity ID matching approach used in the ELR method. When considering the query subsets, we observe that FSDM+ESimcg sig- nificantly outperforms FSDM for SemSearch and QALD queries with respect to NDCG@10, and for INEX-LD queries with respect to NDCG@100. Improve- ments over BM25F-CA were more substantial: BM25F-CA+ESimcg brings sig- nificant improvements for all categories (with respect to all metrics) except for SemSearch queries for NDCG@100.

OVERALL PERFORMANCE. To answer our first first research question, whether embeddings improve the score of entity retrieval, we compare our entity re-ranking approach with a number of baseline entity retrieval models. Table 2 shows the results for different different models with respect to NDCG@10 and NDCG@100, the default evaluation measures for DBpedia-entity V2. In this table, the embedding-based similarity component (Eq. (5)) is denoted by ESim, where c and cg subscripts refer to the two versions of our entity embeddings: without and with link graph. Table 2. Results of embedding-based entity re-ranking approach on different different query subsets of DBpedia-Entity V2 collection. Significance Significance of results is explained in running text. Model SemSearch INEX-LD ListSearch QALD-2 Total NDCG @10 @100 @10 @100 @10 @100 @10 @100 @10 @100 Reranking the FSDM top 1000 entities ESimc 0.365 0.412 0.194 0.252 0.210 0.288 0.192 0.255 0.239 0.300 ESimcg 0.397 0.462 0.216 0.282 0.211 0.311 0.213 0.286 0.258 0.334 FSDM 0.652 0.722 0.421 0.504 0.420 0.495 0.340 0.436 0.452 0.534 +ELR 0.656 0.726 0.435 0.513 0.422 0.496 0.347 0.446 0.459 0.541 +ESimc 0.659 0.725 0.433 0.513 0.432 0.509 0.353 0.447 0.463 0.543 +ESimcg 0.672 0.733 0.440 0.528 0.424 0.507 0.349 0.451 0.465 0.549 Reranking the BM25F-CA top 1000 entities ESimc 0.381 0.424 0.194 0.253 0.211 0.283 0.192 0.252 0.243 0.301 ESimcg 0.417 0.478 0.217 0.286 0.211 0.302 0.212 0.282 0.262 0.335 BM25F-CA 0.628 0.720 0.439 0.530 0.425 0.511 0.369 0.461 0.461 0.551 +ESimc 0.658 0.730 0.462 0.545 0.448 0.529 0.380 0.469 0.481 0.563 +ESimcg 0.660 0.736 0.466 0.552 0.452 0.535 0.390 0.483 0.487 0.572 ± ± The results of our method are presented for components ESimc and ESimcg by themselves (i.e., λ == 1 in Eq. (6)), and also in combination with FSDM and BM25F-CA. The mean and standard deviation of λ found by the Coordinate Ascent algorithm over all folds are: 0.34 0.02 for FSDM+ESimc, 0.61 0.01 for ± ± FSDM+ESimcg, 0.81 0.03 for BM25F-CA+ESimc, and 0.88 0.00 for BM25F- CA+ESimcg. The results show that the embedding-based scores alone do not perform very well, however, when combining them with other scores, the per- formance improves by a large margin. We determine the statistical significance significance of the difference difference in effectiveness effectiveness for both the NDCG@10 and the NDCG@100 values, using the two-tailed paired t-test with α < 0.05. The results show that both versions of FSDM+ESim and BM25-CA+ESim models yield significant significant improvements over FSDM and BM25-CA models (with respect to all metrics), respectively. Also, FSDM+ESimcg improves significantly significantly over FSDM+ELR with respect to NDCG@100, showing that our embedding based method captures entity similarities better than the strong entity ID matching approach used in the ELR method. When considering the query subsets, we observe that FSDM+ESimcg sig- nificantly nificantly outperforms FSDM for SemSearch and QALD queries with respect to NDCG@10, and for INEX-LD queries with respect to NDCG@100. Improve- ments over BM25F-CA were more substantial: BM25F-CA+ESimcg brings sig- nificant nificant improvements for all categories (with respect to all metrics) except for SemSearch queries for NDCG@100.