Architecture variants. A number of different architectures have been tested: • The MobileNetV2 backbone [BB13]. This backbone is used extensively both in 2D and 3D computer vision problems. For an input image of 224 by 224 it has 3.4 million parameters, which is roughly 1/3 of Resnet18 for a similar image size. • A Resnet18 based backbone with weight sharing. Specifically, each image from the focus-stack is processed separately through the backbone after which positional encoding is explicitly added to the feature output vector. This positional information may both be fixed or learned. This architecture may be seen in Figure 66. • Vivit. [BB14] is a vision transformer [BB15] adaptation for video. Here we adapt the architecture to use the channels in the focus-stack as imaginary time component and keep the division into 16 patches per slice. • MobileNet V2 backbones with weight sharing and a GRU (Gated Recurrent Unit) [BB16] layer on top that explicitly learns from the sequence of latent embedded vectors from left to center to right. Version Status Date Page 2.0 Non-Confidential 2024.05.1172022.03.1 76/100 Generally speaking, there isn’t a single architecture that wildly outperforms the others. The weakest performance comes from MobileNetV2 but it is also the least heavy by far. The latter specifically suffers in all states approximately 40nm from the goal state. Otherwise, performance is on comparable. All other architectures manage to achieve over 85% performance on real data. We will revisit this observation extensively in work package 1.2.
