Common use of Query Analysis Clause in Contracts

Query Analysis. Next, we investigate our second research question and analyse queries that are helped and hurt the most by our embedding-based method. Table 3 shows six queries that are affected the most by BM25F-CA+ESimcg compared to BM25F- CA (on NDCG@100). Each of the three queries with highest gains are linked to at least one relevant entity (according to the assessments). The losses can be attributed to various sources of errors. For the query “spring shoe canada”, the only relevant entity belongs to the 2.4% of entities that have no embedding (cf. §4.2). Query “vietnam war movie” is linked to entities Vietnam War and War film, with confidence scores of 0.7 and 0.2, respectively. This emphasizes Vietnam war facts instead of its movies, and could be resolved by improving the accuracy of the entity linker and/or employing a re-ranking approach that is more robust to linking errors. The query “xx xxxxxx fantasy island” is linked to a wrong entity due to a spelling mistake. To conclude, errors in entity linking form one of the main reasons of performance loss in our approach. To further understand the difference between the two versions of the embed- dings at the query-level, we selected the queries with the highest and lowest gain in NDCG@100 (i.e., comparing BM25F+ESimcg and BM25F+ESimc). For the query “Which instruments did Xxxx Xxxxxx play?”, the two linked entities (with the highest confidence score) are Xxxx Xxxxxx and Musical Instru- ments. Their closest entity in graph embedding space is Xxxx Xxxxxx’x musi- cal instruments, relevant to the query. This entity, however, is not among the most similar entities when we consider the context-only case. For the other queries in Table 4, the effect is similar but less large than in the BM25F and BM25F + ESimcg case, probably due to the lower value of λ.

Query Analysis. Next, we investigate our second research question and analyse queries that are helped and hurt the most by our embedding-based method. Table 3 shows six queries that are affected affected the most by BM25F-CA+ESimcg compared to BM25F- CA (on NDCG@100). Each of the three queries with highest gains are linked to at least one relevant entity (according to the assessments). The losses can be attributed to various sources of errors. For the query “spring shoe canada”, the only relevant entity belongs to the 2.4% of entities that have no embedding (cf. §4.2). Query “vietnam war movie” is linked to entities Vietnam War and War film, with confidence confidence scores of 0.7 and 0.2, respectively. This emphasizes Vietnam war facts instead of its movies, and could be resolved by improving the accuracy of the entity linker and/or employing a re-ranking approach that is more robust to linking errors. The query “xx xxxxxx fantasy island” is linked to a wrong entity due to a spelling mistake. To conclude, errors in entity linking form one of the main reasons of performance loss in our approach. To further understand the difference difference between the two versions of the embed- dings at the query-level, we selected the queries with the highest and lowest gain in NDCG@100 (i.e., comparing BM25F+ESimcg and BM25F+ESimc). For the query “Which instruments did Xxxx Xxxxxx play?”, the two linked entities (with the highest confidence confidence score) are Xxxx Xxxxxx and Musical Instru- ments. Their closest entity in graph embedding space is Xxxx Xxxxxx’x musi- cal instruments, relevant to the query. This entity, however, is not among the most similar entities when we consider the context-only case. For the other queries in Table 4, the effect effect is similar but less large than in the BM25F and BM25F + ESimcg case, probably due to the lower value of λ.