Intrinsic evaluations Sample Clauses

Intrinsic evaluations. Intrinsic evaluations focus mainly on word embedding intrinsic qualities as well as how well embedding relations relate to human cognitions. Word semantic similarity test is one of the most widely used intrinsic evaluations. Datasets were collected from human evaluators, who, given pairs of words, were asked to access the similarity of two words within a pair. Similarities calculated by distance within word embeddings of word pairs from the same datasets were compared with the human judgement results. Many datasets have been developed for similarity tests, including WordSim-353 [12] and SimVerb-3500 [15]. Another popular intrinsic evaluation is word analogy test, which was pop- ularized by the word2vec word embedding model [26]. The main assumption is that arithmetic operations on word embeddings can solve analogy problems. Meanly, given four words x,xj, y and yj, if the relationship between x and xj is the same as the relationship between y and yj, it is believed that yj can be predicted by x + xj − y. One popular example would be given the embeddings of King, Man, Queen, then Women embedding can be predicted by king + man - queen. Popular datasets for analogy tests contain Google Analogy [12] and SemEval-2012 [19]. Other intrinsic evaluations include subconscious intrinsic evaluations. Subconscious intrinsic evaluations are evaluations where human cognitive assessments are caught in a subconscious level. For instance, eye movement data evaluations compare the features extracted from the eye movement data and word pair embeddings [44]. Subconscious evaluations also include semantic priming, which is based on the assumption that if a word is preceded by a semantically related word, then a human will tend to read the word faster. Word semantic relations are evaluated based on the time people spent reading the word, given the previous one [7].