The Adjudication Task. We created adjudication pools by adapting the methodology of the National Institute for Standards and Technology (NIST) Text REtrieval Conference (TREC) (Xxxxxxxx and Harman, 2000; Xxxxxxxx, 2001). To create adjudication pools, results were aggregated from several open source and commercial tools using lower-than-normal matching thresholds. To be maximally useful, evaluation should be done with reference to a particular use context. For information retrieval, one consideration is the relative importance of precision and recall, or, put another way, the tolerance for false positives and false negatives. In the use case envisioned for this evaluation, a system presents name search results to a user who then has access to additional identifying attributes to make a decision about an overall identity match. Further, we imagine a scenario in which the cost of a false negative is relatively high. Thus, this user is willing to sift through spurious matches in order to ensure that she does not miss a potentially good identity match. We therefore developed a set of guidelines using a “loose” truth criterion, by which two names should be considered a match despite variation beyond superficial spelling differences, as long as there is a plausible relationship between the names. The guidelines enumerate several types of name variations that can establish such a relationship, including both segment-level variation (e.g. alternate spellings) and structural variation (e.g. additions, deletions, reorderings). For example, the names in Figure 1, in which the data contained in the surname field is capitalized, would be considered a possible match.3 3 Because of the structure of Arabic names, the apparently mismatching elements do not necessarily conflict. Xxx Xxxxx is an optional name element meaning “son of Xxxxx”, Xxxx is an honorific title used by someone who has made the pilgrimage to Mecca, and Xx Xxxxx means “the Egyptian”. It is therefore possible that these two names could belong to a single person whose full name is Xxxx Xxxxxxx Bin Xxxxx Xxxxxxx Xx Xxxxx.
