Last updated: November 21, 2003
Super High Resolution Global Atmospheric Simulation by AFES
This work is a collaboration between the Earth Simulator Center and the Frontier Research System for Global Change.
Introduction

An atmospheric general circulation model called AFES (AGCM for Earth Simulator) was developed and optimized for the architecture of the Earth Simulator (ES). AFES is based on the CCSR/NIES AGCM and is a global three dimensional hydrostatic model using the spectral transform method. We achieved a high sustained performance by the execution of AFES with T1279L96 resolution on the ES. The performance of 26.58 Tflops was achieved the execution of the main time step loop using all 5120 processors (640 nodes) of the ES. This performance corresponds to 64.9% of the theoretical peak performance 40.96 Tflops (Fig. 1). AFES's this performance was recognized as the fastest computation at Super Computing 2002, Baltimore, MD, U.S.A., November 2002, and AFES won Gordon Bell Prize for Peak Performance.
Figure 1. Scalable performance of the AFES. Figure 2. The cost distribution of elapsed time.

Performance Optimization
  1. In order to pursue the best possible performance on the ES, we adopted the three-level parallelization available for the ES, i.e., inter-node parallel processing for distributed memory architecture, intra-node parallel processing for shared memory architecture, and vector processing for vector architecture in a single processor. For inter-node parallel processing, the MPI library is used, and for intra-node parallel processing for the shared memory architecture within each node, microtasking, a kind of thread programming, is used.

  2. The computational complexity of the Legendre Transform (LT) increases accordingly to the third power of the truncation wavenumber, and it becomes dominant in higher resolution. We have especially optimized the LT for vector processing. In fact, Fig. 2 shows that the rate for which the LT accounts to elapsed time has only been about 50 to 60%, and it was smaller than what was expected in the computational complexity. Also shown is that the Legendre transform calculation has very high vector efficiency and a very good scalability in such a super-high resolution.

  3. The ES has a high-speed Interconnection Network (Single-stage full crossbar switch: 12.3GB/s x 2). In order to achieve high-speed communication performance, in transpose communication of spectral transform method, MPI_PUT and MPI_WIN_FENCE which are one-sided communications facility of MPI-2 are used. Furthermore, Global Memory is used. Global Memory, a new type of shared memory, can be shared by multiple processes located in distributed nodes. It can communicate without performing an excessive memory copy with a system buffer. This is also called zero copy.
Simulations

Figure 3 shows a global field of precipitation after seven and a half days of our model integration with the T1279 spectral truncation in horizontal (approximately 10 km x 10 km) and 96 vertical levels (approximately 20 m near surface and 500 m from the middle troposphere to the lower stratosphere). Figure 4 is a magnified picture of Fig. 3 around the Japan area. To the best of our knowledge, no other model simulation of the global atmosphere has ever been performed with such a super-high resolution. Currently, such a simulation is possible only on the ES with our model, AFES. The 10-km grid interval is almost the highest resolution on which the hydrostatic approximation as commonly adopted in AGCMs is valid.

As evident in Fig. 3, meso-scale features that were unresolved in the initial state from reanalysis data with much lower resolution are now emerging in the T1279 simulation, and they appear to be fairly realistic. For example, a moist and precipitating area with a distinctive T-bone shape (Fig. 4) characterizes each of the cyclones developing and migrating eastward over the mid-latitude oceans. This structure apparently consists of a warm frontal zone in east-west orientation and a meridional extending cold front, often observed as narrow, meso-scale structures characteristic of a developing extratropical cyclone.

Moreover, stripe-shaped features, which are commonly seen in satellite pictures of clouds over the Japan area in wintertime, are clearly shown in Fig. 4. The model can also simulate typhoon-like disturbances in the Tropics, some of which can be found in Mov. 1 as heavily precipitating areas spiraling near Japan. To the best of our knowledge, no other global climate model can reproduce tropical cyclones with such realistically looking spiral cloud bands.
Figure 3. Snapshot of global precipitation. Figure 4. Close up of a mid-latitude cyclone.
Quick Time (2.5MB)
MPEG (2.7MB)
Movie 1. Twin typhoons over the Philippine Sea.


Conclusion

Despite its test integration stage, AFES has been proven to run on the ES with remarkably high performance, reproducing some of the meso-scale atmospheric features in a fairly realistic fashion as if they were taken from satellite images. We are, of course, aware of several potential obstacles that must be overcome before a more realistic simulation becomes possible with such a super high resolution. For example, some of the parameterization schemes of physical processes, including convective cloud ensembles and effects of gravity waves, require appropriate adjustments. Handling a huge amount of output data will be a major practical problem. We nevertheless believe that this is certainly the first step towards future global climate modeling with realistic meso-scale processes and topographic effects, with which we may be able to discuss climate and weather characteristics of every region over the globe under the warm future climate.


Publications
  • Ohfuchi, W., T. Enomoto, K. Takaya, and M. K. Yoshioka, 2003: 10-km mesh global atmospheric simulations. Proceedings of the Tenth Workshop on High Performance Computing “Realizing Teracomputing”, European Centre for Medium-Range Weather Forecast, World Scientific, in press.
  • Ohfuchi, W., S. Shingu, H. Fuchigami, and M. Yamada, 2003: The dependence of parallel performance of the atmospheric general circulation model for the Earth Simulator on problem size. NEC Research & Development, 44 (1), 99–103.
  • Ohfuchi, W., S. Shingu, H. Nakamura, M. K. Yoshioka, T. Enomoto, K. Takaya, S. Yamane, T. Nishimura, X. Peng, H. Fuchigami, M. Yamada, Y. Kurihara, and K. Ninomiya, 2003: 10-km mesh meso-scale resolving global simulations of the atmosphere on the Earth Simulator: Preliminary outcomes of AFES (AGCM for the Earth Simulator). To be submitted.
  • Peng, X., F. Xiao, T. Yabe, and K. Tani, 2003: Implementation of the CIP as the advection solver in the MM5. Mon. Wea. Rev., 131 (7), 1256–1271.
  • Peng, X., F. Xiao, W. Ohfuchi, and H. Fuchigami: Conservative semi-Lagrangian transport in a sphere and the impact on vapor circulation in an AGCM, submitted to Mon. Wea. Rev. 2003
  • Shingu, S., H. Fuchigami, M. Yamada, 2003: Vector parallel programming and performance of a spectral atmospheric model on the Earth Simulator. Proceedings of Tenth ECMWF Workshop on High Performance Computing in Meteorology “Realizing TeraComputing”, World Scientific, in press.
  • Shingu, S., H. Fuchigami, M. Yamada, Y. Tsuda, M. Yoshioka, W. Ohfuchi, H. Nakamura, and M. Yokokawa, 2003: Performance of atmospheric general circulation model using the spectral transform method on the Earth Simulator. in Parallel Computational Fluid Dynamics — New Frontiers and Multi-Disciplinary Applications, K. Matsuno et al. (Editors), pp. 79–86, Elsevier.
  • Shingu, S., H. Takahara, H. Fuchigami, M. Yamada, Y. Tsuda, W. Ohfuchi, Y. Sasaki, K. Kobayashi, T. Hagiwara, S. Habata, M. Yokokawa, H. Itoh, and K. Otsuka, 2002: A 26.58 Tflops global atmospheric simulation with the spectral transform method on the Earth Simulator. Proceedings of Supercomputing 2002, http://www.sc-2002.org/paperpdfs/pap.pap331.pdf.
  • Xiao, F., T. Yabe, X. Peng, and H. Kobayashi, 2002: Conservative and oscillation-less atmosphere transport schemes based on rational functions. J. Geophys. Res., 107 (D22), 4609, doi:10.1029/2001JD001532.
  • Yamane, S., M. Honda, H. Nakamura, and W. Ohfuchi, 2002: On interdecadal modulations of prominent variability of boreal winter general circulation field on the Northern Hemisphere appearing in both observational data and numerical simulations —With special attention to the seesaws between the Aleutian and Icelandic lows. Gross Wetter, 40, 88–97 (in Japanese).
  • Yoshioka, M. K., F. Araki, H. Uehara, T. Enomoto, and A. Igarashi: “3-D Structure of Baiu Front around Japan Simulated by Ultra High Resolution Atmospheric General Circulation Model on the Earth Simulator”. Journal of Visualization, Vol.6, No.4., 2003, p.327 (Frontispiecs).

Presentations
  • Enomoto, T.: Representation of the Baiu (Meiyu) frontal zone in a T1279L96 model on the Earth Simulator. International Conference on Mesoscale Convective Systems and Heavy Rainfall/Snowfall in East Asia, Tokyo, Japan, October 2002.
  • Enomoto, T.: A Baiu (Meiyu) frontal zone simulation by a super-high resolution global general circulation model. Society of Marine Research Technology, the Fourteenth Research Conference, Tokyo, Japan, November, 2002 (in Japanese).
  • Enomoto, T.: Simulations of global to regional phenomena using a 10-km mesh AGCM on the Earth Simulator. International Workshop on NWP models for heavy precipitation in Asia and Pacific Areas, Tokyo, Japan, February 2003.
  • Enomoto, T., H. Fuchigami, S. Shingu, and W. Ohfuchi: Representation of the boreal summer general circulation of the atmosphere by AFES with the Emanuel cumulus convection scheme. Meteorological Society of Japan 2002 Fall Meeting, Sapporo, Japan, October 2002 (in Japanese).
  • Enomoto, T., W. Ohfuchi, and K. Ninonimya: Simulation of the Baiu/Meiyu frontal zone using a global 10-km mesh atmospheric model. The 2nd Workshop on Regional Climate Modeling for Monsoon System, Yokohama, Japan, March 2003.
  • Enomoto, T., and W. Ohfuchi: Hindcast simulations of global high-impact weather on the Earth Simulator. Case 1: The August 2002 storms. EGS-AGU-EUG joint assembly, Nice (France), April 2003.
  • Enomoto, T., and W. Ohfuchi: Hindcast simulations of global high-impact weather on the Earth Simulator. Case 2: The November 2002 storms. EGS-AGU-EUG joint assembly, Nice (France), April 2003.
  • Enomoto, T., and W. Ohfuchi: Hindcast simulations of severe weather events by a high-resolution atmospheric general circulation model on the Earth Simulator. Case 2: The November 2002 storms. IUGG, Sapporo (Japan), July 2003., [Poster]
  • Nakamura, H., S. Yamane, M. Honda, and W. Ohfuchi: Interdecadal modulations in the observed and simulated seesaws between the Aleutian and Icelandic lows. American Geophysical Union 2002 Spring Meeting, Washington, DC, U.S.A., May 2002.
  • Ohfuchi, W., T. Nishimura, S. Yamane, and T. Enomoto: The sensitivity of AFES (AGCM for Earth Simulator) to cumulus convective parameterization and resolution. Meteorological Society of Japan 2002 Spring Meeting, Ohmiya, Japan, May 2002 (in Japanese).
  • Ohfuchi, W., T. Nishimura, and S. Yamane: The sensitivity of AFES (AGCM for Earth Simulator) to cumulus convective parameterization and resolution II. Meteorological Society of Japan 2002 Fall Meeting, Sapporo, Japan, October 2002 (in Japanese).
  • Ohfuchi, W., T. Enomoto, K. Takaya, and M. K. Yoshioka: Toward 10-km mesh global climate simulations. American Geophysical Union 2002 Fall Meeting, San Francisco, U.S.A., December 2002.
  • Ohfuchi, W., and T. Enomoto, 2003: Hindcast simulations of global high-impact weaather on the Earth Simulator. Case 1: The August 2002 Storms, EGS-AGU-EUG Joint Meeting, Nice, France, April 2003
  • Ohfuchi, W., and T. Enomoto, 2003: Hindcast simulations of severe weather events by a high-resolution atmospheric general circulation model on the Earth Simulator. Case 1: The August 2002 Storms. IUGG Joint Assembly, Sapporo, June-July, 2003.
  • Ohfuchi, W., S. Shingu, H. Nakamura, M. K. Yoshioka, T. Enomoto, K. Takaya, S. Yamane, T. Nishimura, X. Peng, H. Fuchigami, M. Yamada, Y. Kurihara, and K. Ninomiya 2003: Ultra-high resolution glabal atmospheric simlations. International Conference on Earth System Modelling, Hamburg, Germany, September 2003.
  • Peng, X.: Accurate computation of advection in meteorological model. CAS-Third World Academy-WMO Forum, Shanghai, Peoples' Republic of China, September 2002.
  • Peng, X., F. Xiao, and W. Ohfuchi: Conservative semi-Lagrangian transport of tracers in AFES. Meteorological Society of Japan 2002 Fall Meeting, Sapporo, Japan, October 2002.
  • Peng, X. and F. Xiao, 2003: How much the advection scheme affects climate simulation with AGCM, IUGG2003 (MC11), Sapporo, Japan.
  • Shingu, S., M. K. Yoshioka, H. Fuchigami, and M. Yamada: Numerical experiments on the Earth Simulator with a super-high resolution atmospheric general circulation model (AFES). Meteorological Society of Japan 2002 Spring Meeting, Ohmiya, Japan, May 2002 (in Japanese).
  • Shingu, S., H. Takahara, H. Fuchigami, M. Yamada, Y. Tsuda, W. Ohfuchi, Y. Sasaki, K. Kobayashi, T. Hagiwara, S. Habata, M. Yokokawa, H. Itoh, and K. Otsuka: A 26.58 Tflops global atmospheric simulation with the spectral transform method on the Earth Simulator. SC2002, Baltimore, November 2002.
  • Yamane, S., M. Honda, H. Nakamura, and W. Ohfuchi: On internal and external variances of interannual variability of Aleutian and Icelandic lows. Meteorological Society of Japan 2002 Fall Meeting, Sapporo, Japan, October 2002 (in Japanese).
  • Yamane, S., M. Honda, H. Nakamura, and W. Ohfuchi: On the seasonal predictability under external forcing. Workshop on General Circulation and Medium-Range Weather Forecast, Tokyo, Japan, October 2002 (in Japanese).
  • Yamane, S., H. Nakamura, M. Honda, and W. Ohfuchi, 2003: A simulated seesaw between the Aleutian and Icelandic lows and multi-dacadal modulations. IUGG, Sapporo.
  • Yamane, S., H. Nakamura, M. Honda, and W. Ohfuchi, 2003: Simulated extratropical circulation in northern hemisphere wintertime and its decadal modulation. International conference on earth system modeling, Hamburg.
  • Yoshioka, M. K., S. Shingu, H. Fuchigami, M. Yamada, W. Ohfuchi, and S. Yamane: A management for enormous data output by AGCM (AFES) on the Earth Simulator. Earth and Planetary Science Union 2002 Joint Meeting, Tokyo, Japan, May 2002 (in Japanese).
  • Yoshioka, M. K., Y. Kurihara, W. Ohfuchi, T. Nishimura, and S. Yamane: Climate features during typhoon season in high-resolution AGCM. Meteorological Society of Japan 2002 Fall Meeting, Sapporo, Japan, October 2002 (in Japanese).
  • Yoshioka, M. K., Y. Kurihara, and W. Ohfuchi: Climate simulation for typhoongenesis by high resolution AGCM. American Geophysical Union 2002 Fall Meeting, San Francisco, U.S.A., December 2002.
  • Yoshioka, M. K., Y. Kurihara, and W. Ohfuchi: “A study on mesoscale structure of a tropical cyclone simulated with 10km high resolution AGCM”. EGS-AGU-EUG Joint Assembly 2003, Nice, France, April 2003
  • Yoshioka, M. K., T. Enomoto, and Y. Kurihara: “Typhoon genesis in global high-resolution simulations on the Earth Simulator”. IUGG, Sapporo, JAPAN, June 30 -- July 11, 2003

(C) JAMSTEC / Earth Simulator Center