From the book : PARALLEL CFD (J.H Kwon , A.Ecer , J.Periaux , N.Satofuka ,P.Fox)
Recent years Grid computing continues to challenge the way scientific computing has been done up until a decade ago .Tera-scale computational capacity such as the TERA-GRID project ,grid computing environment by US-NSF ,is now available to researchers for challenging problems . TeraGrid integrated high-performance computers, data resources and tools, and experimental facilities. Resources included more than a petaflop of computing capability and more than 30 petabytes of online and archival data storage, with rapid access and retrieval over high-performance networks computer network connections. Researchers can also access more than 100 discipline-specific databases.
Exept for tera-scale or peta-scale scientists need more advanced tools for CFD high performances such as MPICH-G2 , which is the Grid enabled implementation of the message passing Interface (MPI) . The fact that CFD data blocks are highly inteconnected requires efficient distribution of the blocks such that the communication between the Grid sites will be at the minimum (for a job ) .
The inter-connection between the Grid sites is the challenge for the coders , allocating efficiently the blocks anong Grid sites in order to optimise this communication .
In TeraGrid sites resourse scedulers handle internal distributions of parallel tasks or chunks of dispersed sub-tasks across Grid sites ( giving this static distribution of parallel CFD blocks the above optimization) .
The blocks can be grouped together to ensure optimum distribution of the blocks , hence minimum communication between the Grid sites , applying for example METIS graph partioning libraries for the distribution across the clusters .
Block distribution across Grid sites
A parallel CFD job executes on partitioned data blocks called mesh blocks .As the number of the partioned mesh blocks increases , dependencies of one block to other increases (hence the communication overhead) . The purpose is to achieve the balanced communication between Grid sites so that overall time for the job would be the lowest achievable .
There are several paths to distribute blocks across the Grid sites . One of them is with graph-partitioning (using METIS package ) .Metis offers three different options to decide on the weights , lower the weights more efficient the distribution , to be used as criteria to distribute given blocks across partitions of a given graph .
In partioning the graph , Metis offers to use either of the communication volume betwenn pairs of mesh blocks or mesh block size . It is possible to use combination of block-to-block communication and block size as weights of partioning the graph .
For example the following distribution of the blocks is apllied :
MESH blk#1 <———-weight=100——>MESH blk#2
| GRID site #1 |
————————————————————————
| GRID site #2 |
MESH blk#4 <————-weight=100———->MESH blk#3
Four mesh blocks distributed between two Grid sites based on communication volume with Metis
graph-partitioning .
The problem has 128 and 512 mesh blocks between two Grid sites .Mesh has about 18M tetraedral elements at an airplane surface (512 mesh blocks) .
zeracuse 