Article by Dr. Ryan Honeyager (OBS) and Stephen Herbener (JEDI)
A new major release of the Interface for Observation Data Access (IODA) will occur this month in concert with the next public Joint Effort for Data Assimilation Integration (JEDI) update. IODA is the unique component of JEDI that manages observation data through a separation of concerns. IODA abstracts access to observations, thus insulating scientists from technical aspects such as file formats. Other components of JEDI, such as the forward operator codes and model interfaces, use IODA-provided constructs to select and read observation data, to store intermediate results, and to extract data and diagnostic plots. This release represents a year of dedicated work between multiple groups, including the JEDI and OBS teams at the JCSDA, as well as teams from NOAA’s Environmental Modeling Center (EMC) and the UK Meteorological Office (UKMO). This release represents a key milestone toward our goal of a world-class observation processing system within JEDI.
There were two goals to this release. First, we aimed to produce high-performance data storage containers that can contend with our rapidly growing volume of data. JEDI assimilates data from many sources, including conventional, GNSS, and radiance-based instruments. Second, we aimed to encapsulate IODA’s data storage implementations (file and memory-based) behind a uniform client-facing interface. This encapsulation provides a stable, uniform client interface (IODA Application Programming Interface, aka IODA API) and data model while accommodating a disparate set of formats and file types that JEDI receives from observation data providers.
Figure 1 below shows a high level schematic of the new IODA subsystem. The observation data provider interface is depicted in the top row and illustrates how a new format from a provider can be attached to IODA. The middle row represents the primary component of IODA (IODA-engines) and lists out the pieces of the data model, namely Groups, Variables, Attributes, Dimensions and (data) Types. The data provider interfaces in the top row perform the task of converting the provider format to the IODA-engines data model. The bottom row depicts the client classes in the JEDI system (outside of IODA) that utilize the IODA API.
Figure 1. Schematic of the different IODA-related components entering JEDI in the release (green), in the next quarter (yellow), and in the far future (orange).
The upcoming release (Fig. 1, green boxes) features support for multidimensional variables (e.g., for brightness temperature and radiance data), interfaces for C, C++ and standalone Python, a new file format, and transparent backwards compatibility with the initial release of JEDI.
Post-release plans for new IODA features (Fig. 1, yellow boxes, work in progress) include the addition of direct ingestion of BUFR and ODB/ODC file formats as well as moving IODA converter and diagnostics applications to the IODA API. These features are being jointly developed with JCSDA’s partner agencies.
One of the challenges of a large-scale, operational forecasting system is the ability to access large volumes of data rapidly and efficiently in a heavily parallelized architecture. The new release of IODA considerably improves performance, especially for hyperspectral radiance instruments containing thousands of channels. In alignment with the JCSDA goal of collaborative development, the new developments of IODA (both release and post-release) have demonstrated its facility at enabling partner contributions.