Modeling the Communication Patterns in the Application Sample Clauses

Modeling the Communication Patterns in the Application. In this benchmark test we are mostly concerned with Steps (ii) and (iii) outlined in Figure 3.3 as they have a major impact on the entire computational cost of the application. The finite element matrix assembly phase, Step (ii), is the most embarrassingly parallel kernel in the application since the matrices are first assembled locally and independently by each process. Once local matrices are computed, a synchronization step occurs to collect the local contribution to the global matrix entries relative to interface entities. The amount of data to be transferred during this phase between process i and j is proportional to Mi j. The solution of the algebraic linear system, Step (iii), is more involved and it entails several the solution of the linear system consist, at each iteration, of the following three computational kernels: (1) matrix–vector multiplication, (2) vector inner product, and (3) solution of the preconditioner. From the communication point of view, the matrix–vector multiplication entails the same pattern described in Step (ii), the vector inner product requires an all to all communication (AllReduce), while the application of the preconditioner has a hardly predictable communication pattern due to load repartition algorithms implemented in Trilinos. To be more specific, as preconditioner we use the algebraic multigrid (AMG) solvers implemented in the Trilinos package ML [62]. AMG builds a hierarchy of agglomerated linear systems, whose size decreases as we move from one level to next. In order to balance between computation and communication, if the number of unknowns associated to process p at level l in the hierarchy falls below a given threshold, then the data structures representing the problem at level l are moved to other processes (by using a load re-balancing algorithm) and processor p is left idle during the computations of all levels below l.