Models used. In this work, we used pre-trained models provided by ELMo, Flair and XXXX. We experimented on several ELMo model: model with 2 layers of bidirectional LSTM with a hidden state size of 1028 and an output projection size of 128; model with 2 layers of bidirectional LSTM with a hidden state size of 2048 and an output projection size of 256; model with 2 layers of bidirectional LSTM with a hidden state size of 4096 and an output projection size of 512 [36]. The resulting output embedding size is twice of the output size since the output of forward-direction network and backward-direction are stacked to form the final output embedding, resulting in word embedding dimensions of 256, 512 and 1024. Since the output of ELMo model is a linear combination of three states in the neural network states and our experiments did not contain a downstream task to tune the combination, we tested the output of given by the three states separately and the average of three states output. The three states are the output of the convolutional network, the output of the first LSTM layer and the output of the second LSTM layer. All of the ELMo models were trained on 1 Billion Word Language Model Benchmark which contains approximately 800M tokens crawled from news data [36, 16]. XXXX models with embedding dimension of 768 and 1024 were investigated. The first model has 12 transformer layers with a hidden dimension of 768, and attention head size of 12 and second has 24 transformer layers with a hidden size of 1024 and attention head size of 16 [1]. Both models were trained on Wikipedia (2.5B words) with BookCorpus (800M words) over a long time with 1M update steps [1, 52]. In this work, we used the output of the second to the last layer as the output embedding. Flair models with dimension size of 2048 and 4096 were included in this work as well. Both were trained on a 1-billion word corpus [4] with both being one layer bidirectional LSTM model.
