Capsule Network. The idea of capsules was presented in [45] where cap- sules were proposed as an approach to capture the presence of an entity. A capsule is a set of neurons that collectively produce an activity vector with one element for each neu- xxx to hold that neuron’s instantiation value. The CapsNet model presented in [45] consists of layers of capsules ap- plied to the task of classification of images. Authors for- malise a training procedure based on routing-by-agreement where each capsule makes prediction over the parent cap- sule and computes a coupling coefficient between the ac- tual capsule and the parent capsule outputs. Capsule out- puts are vectors indicating the presence of an entity within the processed input, while their norms represent the con- fidence of the indication. Many works at state of the art, present the application of CapsNet structure to cope with image classification using hyperspectral images [12] and medical images [1, 34, 22]. Works focus on the empower of the CapsNet structure to improve the performance of the model with complex datasets (coloured and real images), in DenseNet [40] authors propose to deep the structure adding dense block to the model, the new layers provide an high level of feature extraction while reducing the model inter- pretability. The idea of stressing out the CapsNet struc- ture to better understand the performances is also presented in [36, 52, 37]. In [40], authors propose two frameworks where the standard convolutional layers is replaced with densely connected convolutions. This helps in extracting feature maps by different layers and it gives a representa- tive base for the primary capsules entities recognition. In our work, we focus on the routing-by-agreement procedure and we propose AA-Caps , a modified CapsNet, based on self-attention mechanism as a valid alternative to routing- by-agreement algorithm. The attention mechanism learns from the votes computed by the primary capsule layer, a compatibility function between low-level features and high- level features. Where the compatibility function is a feed forward neural network with a softmax activation function, the transformers presented in [49] were the first implemen- tation of attention function. We think that the CapsNet needs a smarter and lighter mechanism for training, and, in- stead of modifying the architecture by a deeper structure, we propose to remove the iterative mechanism of learn- ing with a non-iterative one. This modification makes th...
