Article by Dr. Mark Olah, JCSDA JEDI core team
This month the JCSDA is announcing a new Near-Real Time (NRT) observation monitoring web product based on the Joint Effort for Data assimilation Integration (JEDI). This product represents the achievement of several firsts for the JCSDA core team as we near our scheduled initial release of JEDI software.
With the launch of the JEDI NRT web product, the JCSDA is proud to report the following accomplishments:
The JCSDA is now regularly ingesting observation data for the complete set of operational instruments at NOAA. These observations are converted into the IODA data format, and are being made available to JCSDA researchers on several HPC platforms via new distributed, decentralized data management and transfer facilities.
NOAA operational Global Forecasting System (GFS) model forecasts are regularly reprocessed, re-gridded at various resolutions, and made available to JCSDA researchers.
JEDI data assimilation software libraries and their interface with the GFS operational model are now being continually tested on most of NOAA’s operational instruments with more to follow.
All of these functionalities along with their runtime management, logging, distributed data management, user interface, and website generation technology are provided by the new JEDI-Rapids workflow management system.
The JEDI workflow system, JEDI-Rapids, makes it possible to implement many different JEDI-based data assimilation applications. It consists of two parts: (1) a set of generic, composable applications corresponding to JEDI executables and their associated data, and (2) a generic workflow-generator allowing customizable generation of graphical workflow structures that combine individual JEDI applications into full data assimilation suites. As a workflow-generation system, JEDI-Rapids is designed to programmatically produce concrete workflow descriptions for a range of production-quality workflow management software engines including ecFlow, Cylc, and Apache Airflow. The workflow management engines are then responsible for actually scheduling and monitoring the execution of applications and executables.
Figure 1. A simplified graphical representation of the workflow graph that implements the NRT website. Each node in the graph is a JEDI-Rapids App, which can be used independently or as part of a larger workflow.
The JEDI Near-Real Time (NRT) observation monitoring website serves as both a development platform for workflow and resource management and scheduling, but also is a valuable testing tool. While the JEDI system has been designed with extensive self-testing, and utilizes continuous integration (CI) for automation of testing at multiple levels, all of these tests are based on processing fixed data samples. Real world, live-data is much more difficult to handle. Running a continuously cycling observation and model data ingest process helps us to find the types of bugs that would be impossible to replicate with static data. Hence, the initial objectives of the NRT system are less focused on scientific fidelity of resulting analysis, and more focused on testing and development.
Figure 2. The JEDI NRT Observation monitoring site is updated at 6hr intervals. All operational GSI instruments with functional Unified Forward Operator (UFO) implementations are run through the system, and animated plots of locations and observation quality metrics are available.