Ricardo Todling1, F. Hebert4, A. Shlyaeva4, D. Holdaway2, B. Menétrier3, Y. Tremolet4, F.R. Diniz4, W. Gu5, J. Jin5, A. Sewnath5, M. Sienkiewicz5, D. Ardag5, Y. Zhu1, R. Gelaro1, M. Wlasak6, M. Destouches6, J. Colclough6, O. Lomax6, H.G. Arango7
(1) NASA/GMAO, (2) NOAA/NCEP, (3) Norwegian Meteorol. Institute, (4) JCSDA, (5) SSAI/GMAO, (6) UK Met Office, (7) Rutgers University
Abstract
In the past couple of months the JEDI effort has incorporate two fixes that amount to a world of difference in the process of transitioning GEOS from the Gridpoint Statistical Interpolation (GSI) analysis to JEDI. The first fix tackles mysteries that have persisted in some of the results obtained with the minimizers and consistencies associated with weak reproducibility. The second fix tackles the ability to use (hybrid) 4DEnVar in the context of employing the GSI background error covariance (GSIBEC) formulation in JEDI. This brief report provides illustrations of the original issue with the minimizers and combines its fix to that associated with the 4D GSIBEC's capability to show preliminary results with non-cycling hybrid 4DEnVar using GEOS background fields.
I. HALO HANDLING IN ATLAS INTERPOLATORS
Without getting deep into the multiple JEDI-github issues, discussions and pull requests associated with problems found with the halo exchange in JEDI in connection to the Altas field-sets, a rough description of the problem and its solution follows. Not too long ago, it has been determined that the current JEDI data flow for the generic data in Atlas had two issues: (i) the toFieldSet method includes a halo exchange, which means that mathematically some desirable properties are not satisfied; and (ii) the recommendation to always provide up-to-date halos is problematic because it requires more halo exchanges (and adjoints) than strictly necessary, and in particular, it is not always clear where they should go, especially in adjoint code. Two examples of desired properties that are not satisfied include: the desire for fromFieldSet to be the inverse of toFieldSet; and the desire for fromFieldSetAD to act the same as toFieldSet (and toFieldSetAD to act as fromFieldSet), but the additional halo exchanges make these operations act differently to each other. The ultimate fix ended up: (a) separating the halo exchange from the toFieldSet and fromFieldSet methods; and (b) making it the responsibility of any code using the halo points to first call a halo exchange to synchronize the halo data; at other points in the code, the halos should be assumed to be unsynchronized.
One of the key consequences of the halo issues above is the lack of weak reproducibility of the solution obtained by any of the minimizers available in JEDI. This affects certain configurations of JEDI, particularly those relying on using the Gridpoint Statistical Interpolation (GSI) Background Error Covariance (GSBEC) formulation to assimilate observations using finite-volume cubed-grid interface to, for example, GEOS fields. Figure 1 shows the behavior of the observation cost function (left) and corresponding gradient reduction (right) for a 1o (C90) 3DVAR configuration using the so-called Derber-Rosati Preconditioned Conjugate Gradient (DRPCG) available in OOPS. Two MPI configurations associated with the horizontal grid distribution have been chosen to solve the same problem, one uses a 10 × 6 distribution layout, and another uses a 6 × 6 layout. Without the halo fixes the observation cost function and the gradient reduction end up being completely different between the two layout configurations (blue curves). The halo issue even causes the convergence rates to be different to the point where in one case (6 × 6) the minimization reaches the prescribed convergence criterium sooner than in the other case (10 × 6); thus iterating less than the maximum number of 50 allowed iterations. When the halo exchange issues are tackled the rate of convergence is very similar (right) leading to observation cost functions (left) laying almost on top of each other.
Fig. 1. Behavior of DRPCG before and after halo fix using two different number of processors configuration for MPI distribution.
As it can be imagined, the different convergence rate and resulting fit to the observations (cost function) lead to quite different solutions. The illustration here involves a simple case, when only radiosondes are used as observations. Even in this case, the difference in the incremental solution when the halo issue is present is of comparable size to the increment itself. Figure 2 shows the difference in zonal wind increment at 200 hPa for the two solutions obtained before resolving the halo issue (top) and after (bottom). Although with the halo fix the solution is not fully reproducible when different layouts are chosen, the closeness is much more acceptable and far from being near the magnitude of the increments themselves.
Fig 2. Comparison of increment between two MPI configurations before (top) and after (bottom) halo fix.
II. NON-CYCLING GEOS-JEDI HYBRID 4DENVAR USING GSIBEC
It has been reported elsewhere that JEDI has been interfaced with the code implementing the background error covariance formulation used in GSI (GSIBEC). The idea is to allow for close comparison between JEDI results and those from the currently operational GSI without having to focus on retuning a different formulation of these errors in JEDI (as it would be required if using the Background error on an Unstructured Mesh Package; BUMP). In principle, since the GSIBEC software (extracted and wrapped from GSI) carries the climatological and ensemble background error capabilities of GSI, it should be possible to exercise GSI’s formulation of 3DVAR, hybrid 3DVAR and hybrid 4DEnVar within JEDI. Although this has been the case for the first two configurations, the latter 4D option has been difficult to exercise in JEDI. A number of months ago the JEDI 4DEnVar cost function (supporting both hybrid and pure ensemble) went through a needed revision (and some corrections). Even with those, the fact that the JEDI 4DEnVar cost function parallelizes the 4D window in time makes interfacing with GSIBEC somewhat complicated. The fact is that GSI does not parallelize its 4D operations, therefore, a further revision was introduced to the System Agnostic Background Error Representation (SABER; the interfacing software) that allows for GSIBEC to bypass the time-parallelization, receive a handle to the 4D-increment, and be used as originally designed. With this change, it is now possible to exercise hybrid 4DEnVar in JEDI using the same hybrid background error covariance employed by GSI.
A brief illustration of the GSIBEC-based JEDI 4D capability is provided in Fig. 3. This is the result of a very simple test done during implementation and testing assimilating a single 1000 hPa zonal wind observation. The case is setup with only a 9-member GEOS ensemble and uses pure (non- hybrid) 4DEnVar. The snap shots in the figure are for the corresponding temperature increment as it moves along in time. The observation is placed in the middle of the time window, and the pictures show the expected propagation of the increment within the hourly instances of the analysis.
Fig 3. Increment of temperature at lowest model level due to single zonal wind observation at [135W,38S,1000hPa] taken in the middle of the 6-hour assimilation window. This is a simple illustrative case (not a scientific evaluation) with only a nine member ensemble and a pure 4DEnVar configuration (not hybrid) using GEOS ensemble and backgrounds.
Clearly, simple single observation tests are made for sanity check. A true test requires use of more convincing observ- ing systems. Fortunately, tremendous progress has also been made along the lines of enabling JEDI’s Unified Observation Operator (UFO) to use all the types of observations handled in GSI with similar error prescriptions and quality control knobs. For the first time now it is possible to start making very close comparisons of JEDI results with those of GSI using almost all its available knobs. Two futures still not available in JEDI are the Tangent Linear Normal Mode Constraint (TLNMC) for initialization purposes and a dry-mass constraint that ensures dry mass remains nearly constant before and after the analysis. Therefore, the results discussed here have these features turned off when it comes to GSI. Table I shows the configuration of the JEDI experiment, and particularly the observing system usage, highlighting that all observations used in the control GSI are now used in JEDI — readers should also contrast this with similar table shown in a previous report1. Not only does the JEDI experiment here include all the features listed in table, but it is also made meaningful by benefiting from the halo-fix discussed in the previous section.
Before we examine some of the results from running GSIBEC-based JEDI it is worth pointing out that comparing results between GSI and JEDI can be tricky. Scalars like cost and gradient reduction are relatively straightforward to compare (although attention is given to a 0.5 factor missing in GSI’s cost function), but comparing the increments is less so. However, GSI operates on a regular-grid, whereas JEDI operates on a cubed-grid. In the tests here, GSI runs at 0.5o (361 × 576, lat-lon grid), whereas the two JEDI experiments are performed at C180 and C90, approximately, 0.5o and 1o, respectively. Therefore, when increments are compared (differed), they are interpolated to the corresponding coarser grid.
Figure 4 shows the behavior of the cost function (both fit to observations and fit to backgrounds) and the gradient reduction for the control GSI with most of its “bells and whistles” and a corresponding hybrid 4DEnVar using JEDI. Only the C180 JEDI experiment results are compared with GSI here. The fit to the background (Jb; left) is very close between the two experiments; there are still some differences in the fit to the observations (Jo; left). These differences can be related to a number of things including, and most likely the fact that GSI uses a dual resolution climatological/ensemble covariance formulation whereas GSIBEC in JEDI is not yet capable of using a similar formulation. The gradient behavior is also rather similar, especially for the inner iterations of the first outer loop; results for the second outer loop are different; JEDI’s result looks somewhat odd and needs further investigation.
Fig 4. Behavior of DRPCG in JEDI (blue) and GSI (red). Top: fit to observations (Jo) and fit to backgrounds (Jb). Bottom: logarithm of gradient reduction. Only C180 JEDI experiment is compared to 0.5 degree GSI.
Lastly, Fig. 5 shows the difference of temperature increments at 500 hPa of the two C90 (top) and C180 (bottom) JEDI runs and the corresponding GSI control. Clearly, resolution matters, and indeed the underlying differences in handling the climatological and ensemble terms (as mentioned above) likely play a major role in the differences seen here. Making the two analyses work on similar grids (although not identical) helps reduce differences in results; treating the covariance formulation consistently is sure to reduce these differences further (this is work to be done). Other fields at other levels compare similarly, although there are some larger than desir- able differences spotted in the stratosphere that will need close understanding (not shown).
Fig. 5. Difference of 500 hPa temperature increment for two JEDI experiments, C90 (top) and C180 (bottom) as compared against GSI's corresponding result. Increment at center time of hybrid 4DEnVar 6-hour window.
III. CLOSING REMARKS
This brief report provides illustrations from recent correc- tions and revised implementations that amount to a world of difference in bringing a GSIBEC-based JEDI system closer to becoming ready for a head-to-head cycling comparison with a corresponding GSI-based system. The illustrations here have all focused on GEOS, but should translate to GFS just as well. Specifically, this note illustrates the consequence of fixes to the halo exchange associated with the JEDI interpolators and a further revision to the knob that supports 4DEnVar applications using the GSI Background Error Covariance formulation. The first allows JEDI to better support weak reproducibility requirements. The second allows JEDI to run in the configuration used operationally at GMAO.
When the halo fixes are combined with the recent 4D revision, the results of JEDI compare rather well with GSI’s for a first try, particularly when using an observing system that is now one-to-one with GSI’s. This initial exercise with near full capability highlights issues that will need to be looked at more closely in the coming months. Specifically, the second outer iteration in JEDI seems to show suspicious behavior. Furthermore, we have found that the diagnosed cost function of JEDI does not seem to correctly account for the correlated observation errors specified in the minimization, and lastly but not least close examination of the behavior of the variational bias correction needs to conducted. These are all part of work that needs to be done to complete standalone comparisons with the current system. The key aspect here is to emphasize the fact that it is the halo fix introduced recently that now opens to door for these assessments to be made reliably.
Acknowledgment: The work performed here would not have been possible without the tremendous achievements of D. F. Parrish, J. C. Derber, and R. J. Purser, from NOAA/NCEP, in implementing the background error covariance machinery in GSI.
Banner image by Tom Barrett on Unsplash