Introduce OpenMP for threads on cores and OpenACC for GPUs Sample Clauses

Introduce OpenMP for threads on cores and OpenACC for GPUs. A code which uses both MPI and OpenMP, or a code that uses both MPI and OpenACC, is referred to as a mixed-mode code. This is done for two reasons: reduce memory footprint or/and speed up application. Not all MPI codes benefit from becoming mixed-mode codes. The benefits are as follows. • Hybrid applications have a reduced memory footprint (the shared memory model allows threads to avoid halo regions or ghost cells) • Eases load balance issue (usually the complexity of (adaptive) load balance growths with the number of subdomains) • Load balancing in threads is much easier o thread-pool model, or o tasks • For applications which are MPI-bound due to load imbalance (long barriers in MPI_Wait or MPI_Receive/Send), it might be advisable to reduce the number of processes and increase the threads, while using OpenMP’s built in load balancing features Whilst the drawbacks are as follows: • In case of MPI_THREAD_MULTIPLE, the application might lose portability o forked threads are allowed to call any MPI routines • Shared memory applications have their own problems o e.g. false sharing, where a cache line is voided repeatedly § this is naturally avoided by MPI processes • NUMA effects, e.g. where the data is placed in memory o this can be resolved by careful task mapping. Maybe also better to use OpenMP 4.5 target directives than OpenACC.