Capsules and Attention Heads Sample Clauses

Capsules and Attention Heads. In networks base on capsules, each pair of capsules types from two adjacent layers has a different transformation ma- trix. So each instantiation of different capsules types has a different point of view over the capsules at the previous layer. In par with this, in a self-attention layer, there are multiple attention heads and each attention head uses a set of transformation matrices to obtain the projection of key, value, and query. So, each attention heads works on a dif- ferent projection of the representations in the lower layer. The purpose of the two mechanisms is to have different ker- xxxx in convolutional neural networks in order to have dif- ferent representations/points of view. Capsules with differ- ent types have a different point of view, but at the lower layer, the assignment probabilities for a capsule are nor- malised over all capsules in the higher layer regardless of the type. Hence we have one assignment distribution per capsule in the lower layer. In the self-attention layer, each head processes independently its input, with the result of a separate attention distribution for each position in the higher layer. The outputs of the heads are only combined at the last step where they are simply concatenated and linearly transformed to compute the final output of the multi-headed attention block.