Record Linkage - Entity Resolution. Record linkage (RL) is the process of locating records in a dataset referring to the same entity across different data sources. RL is necessary when joining datasets based on entities that may or may not share a common identifier (Christen; Herzog, Xxxxxxxx, and Xxxxxxx). In data warehousing, RL is fundamental: each source system may have a different way of representing same entities. Usually, RL occurs in the second phase of the ETL process. Entity resolution (Xxxxxxxx) is a process executed by a middleware, where non-obvious relationships across different data silos can be exposed, allowing the connection of such sources. Entity resolution engines apply rules, based on deterministic or probabilistic rules, to identify hidden relationships across the data. The simplest kind of record linkage, called deterministic or rules-based record linkage, generates links based on the number of individual identifiers matching among the available datasets (Xxxx and Xxxxx). When all of some identifiers (above a threshold) are identical, two records are considered matching. This approach is ideal when entities share common identifiers, and when quality of data is relatively high. More complex deterministic rules have been devised to infer more complicated connections (Xxxxxxxx; Xxxxxx and Xxxxxxxxxxxxxxx; Herzog, Xxxxxxxx, and Xxxxxxx). When data quality decreases or data complexity increases, the number of deterministic rules needed rapidly grows, making the usage of specialized software tools (see Section 3) fundamental. Moreover, when new data, with characteristics different than expected when devising the rules, enter the system there could be the need of a complete restructure of the deterministic rules. Probabilistic record linkage (Xxxxxxx), or fuzzy matching, probabilistic merging, has a different approach: considering a large range of identifiers, estimating the ability to identify a match or non-match for each identifier, building a weighted set, and using these weights to estimate the probability that two records refers to the same entity. This approach usually require a training phase, that can use a set of gold examples manually entered in the system (and in modern systems, this phase can be carried out via crowdsourcing).
