QA Systems. It has become a common understanding that RNN (Recurrent Neural Net- works), CNN (Convolutional Neural Network) and miscellaneous attention mechanism are required to ensure competitive performance on QA datasets. A lot of neural models have been presented to demonstrate remarkable results on the datasets in Section 2.1. R-Net [30] uses gated attention-based recur- rent networks and refines QA representation with self-matching attention. ReasoNet [26] takes multiple turns to reason over the relationships between query, documents, and answers. Attention Over Attention Reader [4] is designed to better capture the similarities between questions and answer contents. Reinforced Mnemonic Reader [9] combines the memorized atten- tion with new attention. Self-attention [29] is solely applied to Question Answering tasks, which became known as the Transformer. Multi-layer Em- bedding with Memory Network (MEMEN) [20] captures better embeddings and document-query pair information. FusionNet [10] keeps the history of word representations and used multi-level attention. Novel and rich contextu- alized word representations[25] is also used with a standard neural architecture instead of focusing on the interaction between document and questions like other works do. Stochastic Answer Network (SAN) [14] utilizes a stochastic prediction dropout layer as the final layer. QANet [34] is constructed with both CNN and self-attention to combine local and global interactions. Em- beddings from Language Models (ELMo) [21] extracts hidden states within bi-directional LSTMs and Bidirectional Encoder Representations (XXXX) [5] uses deep-layered transformers to generate contextualized word embeddings.
