Convolutional Neural Networks Sample Clauses

Convolutional Neural Networks. ‌ The designed convolutional neural network model is motivated by [161]. First, a convolutional layer is applied to the image of text using the hy- perbolic tangent activation function. The image which is an input consists of rows standing for consecutive words in two sentences, the question (q) and the answer candidate (a), where the words are represented by their embed- dings [89]. In the experiments, the images of 80 rows (40 for question and answer, respectively) are used. If any of the question or answer is longer than 40 tokens, the rest is being cut from the input. Next, the max pooling is ap- plied to the feature maps and the sentence vectors for q and a are generated. Experimentation with the average pooling as [161] has led a marginally lower accuracy thus the max pooling is used in the final version of the framework. Unlike [161] who performed the dot product between these two vectors, this work adds another hidden layer and learn their weights. The framework supports retraining word embedding to which has not been done previously. Finally, the sigmoid activation function is applied and the entire network is trained using the binary cross-entropy. Next, a logistic regression model is applied, where the convolution neural network score is used as one of the features. Other features in the logistic regression are the number of overlapping words between q and a, say Λ, Λ normalized by the IDF, and the question length. While the logistic regression model could be merged directly with designed convolutional neural network model, it has been empirically shown that it is more effective to construct this last phase as a separate model [171]. While a neural architecture is extremely useful to extract hidden semantic structure, it lacks the skill to DENSE <0;1> VA VQ A Q POOL CONV INPUT embedding Subtree matching LR WO WO IDF LEN P words Figure 3.4: The overview of the system that uses a convolutional neural network and logistic regression. expose a simple lexical matching, which often is an important signal for the final classifier. The training phase is a separate process for both convolutional neural net- work model and logistic regression model. First, the convolutional model is trained using early stopping method on development split. After the model is trained, its prediction scores on training, development and test splits are extracted and used in a separate training for logistic regression. For the fu- ture predictions, this framework runs as a pi...

Convolutional Neural Networks. ‌ We will use Kim’s CNN model as the baseline CNN. In this CNN model [5], every word is represented using a word vector and a document can therefore be represented by a matrix. We then use filters of lengths 2, 3, 4 to extract features related to 2,3,4 grams from the text. We concatenate the features together to form a new matrix, which represents the original text in more local details. In the following step, we will perform the global max pooling operation along the vertical axis to extract the most salient features from each dimension space, forming a document embedding with the same dimensions as the word embeddings. This model is merely for baseline establishment, so we do not expect the model to completely outperform the traditional state-of-art models. A conceptual understanding of Kim’s CNN can be seen in Fig 4.2. Figure 4.2: An overview of CNN [5]