Updated: June 1, 2004
Super High Resolution Global Atmospheric Simulation by AFES
This work is a collaboration between the Earth Simulator Center and the Frontier Research System for Global Change.
Introduction
An atmospheric general circulation model called AFES (AGCM for Earth Simulator) was developed and optimized for the architecture of the Earth Simulator (ES). AFES is based on the CCSR/NIES AGCM and is a global three dimensional hydrostatic model using the spectral transform method. We achieved a high sustained performance by the execution of AFES with T1279L96 resolution on the ES. The performance of 26.58 Tflops was achieved the execution of the main time step loop using all 5120 processors (640 nodes) of the ES. This performance corresponds to 64.9% of the theoretical peak performance 40.96 Tflops (Fig. 1). AFES's this performance was recognized as the fastest computation at Super Computing 2002, Baltimore, MD, U.S.A., November 2002, and AFES won Gordon Bell Prize for Peak Performance.
 |
 |
| Figure 1. Scalable performance of the AFES. | Figure 2. The cost distribution of elapsed time. |
Performance Optimization
In order to pursue the best possible performance on the ES, we adopted the three-level parallelization available for the ES, i.e., inter-node parallel processing for distributed memory architecture, intra-node parallel processing for shared memory architecture, and vector processing for vector architecture in a single processor. For inter-node parallel processing, the MPI library is used, and for intra-node parallel processing for the shared memory architecture within each node, microtasking, a kind of thread programming, is used.
The computational complexity of the Legendre Transform (LT) increases accordingly to the third power of the truncation wavenumber, and it becomes dominant in higher resolution. We have especially optimized the LT for vector processing. In fact, Fig. 2 shows that the rate for which the LT accounts to elapsed time has only been about 50 to 60%, and it was smaller than what was expected in the computational complexity. Also shown is that the Legendre transform calculation has very high vector efficiency and a very good scalability in such a super-high resolution.
The ES has a high-speed Interconnection Network (Single-stage full crossbar switch: 12.3GB/s x 2). In order to achieve high-speed communication performance, in transpose communication of spectral transform method, MPI_PUT and MPI_WIN_FENCE which are one-sided communications facility of MPI-2 are used. Furthermore, Global Memory is used. Global Memory, a new type of shared memory, can be shared by multiple processes located in distributed nodes. It can communicate without performing an excessive memory copy with a system buffer. This is also called zero copy.
Simulations
Figure 3 shows a global field of precipitation after seven and a half days of our model integration with the T1279 spectral truncation in horizontal (approximately 10 km x 10 km) and 96 vertical levels (approximately 20 m near surface and 500 m from the middle troposphere to the lower stratosphere). Figure 4 is a magnified picture of Fig. 3 around the Japan area. To the best of our knowledge, no other model simulation of the global atmosphere has ever been performed with such a super-high resolution. Currently, such a simulation is possible only on the ES with our model, AFES. The 10-km grid interval is almost the highest resolution on which the hydrostatic approximation as commonly adopted in AGCMs is valid.
As evident in Fig. 3, meso-scale features that were unresolved in the initial state from reanalysis data with much lower resolution are now emerging in the T1279 simulation, and they appear to be fairly realistic. For example, a moist and precipitating area with a distinctive T-bone shape (Fig. 4) characterizes each of the cyclones developing and migrating eastward over the mid-latitude oceans. This structure apparently consists of a warm frontal zone in east-west orientation and a meridional extending cold front, often observed as narrow, meso-scale structures characteristic of a developing extratropical cyclone.
Moreover, stripe-shaped features, which are commonly seen in satellite pictures of clouds over the Japan area in wintertime, are clearly shown in Fig. 4. The model can also simulate typhoon-like disturbances in the Tropics, some of which can be found in Mov. 1 as heavily precipitating areas spiraling near Japan. To the best of our knowledge, no other global climate model can reproduce tropical cyclones with such realistically looking spiral cloud bands.
 |
 |
||||
| Figure 3. Snapshot of global precipitation. | Figure 4. Close up of a mid-latitude cyclone. | ||||
 |
|
||||
| Movie 1. Twin typhoons over the Philippine Sea. | |||||
Conclusion
Despite its test integration stage, AFES has been proven to run on the ES with remarkably high performance, reproducing some of the meso-scale atmospheric features in a fairly realistic fashion as if they were taken from satellite images. We are, of course, aware of several potential obstacles that must be overcome before a more realistic simulation becomes possible with such a super high resolution. For example, some of the parameterization schemes of physical processes, including convective cloud ensembles and effects of gravity waves, require appropriate adjustments. Handling a huge amount of output data will be a major practical problem. We nevertheless believe that this is certainly the first step towards future global climate modeling with realistic meso-scale processes and topographic effects, with which we may be able to discuss climate and weather characteristics of every region over the globe under the warm future climate.