Convolutional Neural Networks. DL architecture, known as convolutional neural networks (CNNs), is the technique of choice for computer vision tasks and the most successful type of model for image analysis (18,300). Their advantage over the multi-layer XXX lies in their robustness to variations in shape, location, and position of regions of interest in images (275). CNNs (see Figure 2-12) are typically composed of three key layers that maintain the spatial relationship in data, with each layer operating on a specific region of the previous layer until the final output. Firstly, the stack of several convolution layers performs feature extraction through the operations of kernels (or filters), which are small arrays of numbers, usually of size 3 x 3, applied across the input to detect specific features wherever they occur in that layer. The convolution function involves this array of numbers or ‘weights’ in a kernel which are multiplied by the array of numbers within the input creating an output value. As the kernels systematically move along small patches of the input image, and the specific feature to be detected is present, this convolution operation continues throughout, generating a feature map. The distance covered over the input between two successive kernel positions is known as the stride and is an adjustable parameter for computational efficiency if we want to minimise the dimensionality of the data. Multiple different filters are applied and generate several feature maps highlighting different characteristics of the input image. Features derived in one layer are passed onto subsequent convolution layers and convolved with a different set of filters to generate more abstract features. The feature maps are passed through non-linear activation functions to ensure only relevant features are activated and passed onto subsequent layers with redundant features reduced. The second key layers are the pooling layers, typically applied after the convolution layers. This layer involves an operation resulting in the simplification of the output from a convolution layer by computing the maximum or average function of the region. This process is a form of downsampling, by reducing output size and giving translational invariance to shifts or rotations in the image so that the relevant information that has been extracted is maintained, whilst the number of parameters is minimised. After these first two layers concerning the multilayer, hierarchy extraction of relevant features, the final convolu...
Convolutional Neural Networks. The designed convolutional neural network model is motivated by [161]. First, a convolutional layer is applied to the image of text using the hy- perbolic tangent activation function. The image which is an input consists of rows standing for consecutive words in two sentences, the question (q) and the answer candidate (a), where the words are represented by their embed- dings [89]. In the experiments, the images of 80 rows (40 for question and answer, respectively) are used. If any of the question or answer is longer than 40 tokens, the rest is being cut from the input. Next, the max pooling is ap- plied to the feature maps and the sentence vectors for q and a are generated. Experimentation with the average pooling as [161] has led a marginally lower accuracy thus the max pooling is used in the final version of the framework. Unlike [161] who performed the dot product between these two vectors, this work adds another hidden layer and learn their weights. The framework supports retraining word embedding to which has not been done previously. Finally, the sigmoid activation function is applied and the entire network is trained using the binary cross-entropy. Next, a logistic regression model is applied, where the convolution neural network score is used as one of the features. Other features in the logistic regression are the number of overlapping words between q and a, say Λ, Λ normalized by the IDF, and the question length. While the logistic regression model could be merged directly with designed convolutional neural network model, it has been empirically shown that it is more effective to construct this last phase as a separate model [171]. While a neural architecture is extremely useful to extract hidden semantic structure, it lacks the skill to DENSE <0;1> VA VQ POOL CONV INPUT embedding Subtree matching WO WO IDF LEN words Figure 3.4: The overview of the system that uses a convolutional neural network and logistic regression. expose a simple lexical matching, which often is an important signal for the final classifier. The training phase is a separate process for both convolutional neural net- work model and logistic regression model. First, the convolutional model is trained using early stopping method on development split. After the model is trained, its prediction scores on training, development and test splits are extracted and used in a separate training for logistic regression. For the fu- ture predictions, this framework runs as a pipeline pe...
Convolutional Neural Networks. We will use Kim’s CNN model as the baseline CNN. In this CNN model [5], every word is represented using a word vector and a document can therefore be represented by a matrix. We then use filters of lengths 2, 3, 4 to extract features related to 2,3,4 grams from the text. We concatenate the features together to form a new matrix, which represents the original text in more local details. In the following step, we will perform the global max pooling operation along the vertical axis to extract the most salient features from each dimension space, forming a document embedding with the same dimensions as the word embeddings. This model is merely for baseline establishment, so we do not expect the model to completely outperform the traditional state-of-art models. A conceptual understanding of Kim’s CNN can be seen in Fig 4.2. Figure 4.2: An overview of CNN [5]
