Article by Stephen Herbener, JCSDA JEDI Core Team
The Interface for Observation Data Access (IODA) is a component of the Joint Effort for Data assimilation Integration (JEDI) software system. From a user’s perspective, IODA handles the input of observation data and output of diagnostic data. Efforts to transform IODA from a prototype system into an operational system that can handle big data is under way, and the task of collecting requirements from the JCSDA partner organizations was initiated by the 2019 IODA Workshop. The workshop took place from February 11 through February 13, 2019 in Monterey, California, and was attended by representatives from the U.S. Navy, NOAA, NASA, ECMWF, UK Met Office, NCAR, UCAR and JCSDA. In total 41 people attended from 8 partner organizations.
The workshop opened on Monday (February 11th) with introductory remarks from two of the organizers Nancy Baker (NRL) and Yannick Trémolet (JCSDA). These talks were followed by speakers from the Navy, NOAA, NASA, ECMWF and the UK Met Office describing requirements for observation handling from their various perspectives. The opening presentations were very informative, and provided useful requirements information for IODA along with an effective foundation for the focused discussions that were to take place through the remainder of the workshop.
Three focused discussion sessions were conducted over the remainder of the workshop. Tuesday’s topics were “in-memory observation handling” and “File and I/O requirements for observation handling”. The third discussion session took place on Wednesday with the subject “pre- and post-processing diagnostics requirements”. Each discussion session was preceded by a set of talks given by representatives from the various partner organizations which served to prepare everyone for the associated discussion. Each set of talks were designed to allow the audience to hear from multiple perspectives including users of DA systems, scientist contributors to JEDI, and JEDI software developers. Material from the presentations was condensed, on the fly, into a list of prompts that were used to seed the subsequent discussion. All workshop participants attended all of the discussion sessions resulting in an effective means for communicating requirements and perspectives among the different partner organizations in the JEDI project.
Work is under way to compile the requirements gathered at the workshop and transform these into the necessary actions to satisfy them. The compilation step has yielded categories that represent different aspects of the IODA design. These categories include themes such as security (the need to handle classified and private data), reliability (an operational system needs to be running 24/7), ease of use, and portability (runs with a variety of hardware platforms, software compilers, and programming languages). We will be using agile software development practices, such as ZenHub tracking and code sprints, to implement solutions for the IODA requirements. The results of the compilation of requirements along with the tracking of the work will be posted where they can be viewed by the partner organizations.
Overall, the workshop was conducted in a highly positive and enthusiastic manner that solidly reflected the collaborative spirit of the JEDI project. People from different organizations showed up with the intention of shaping IODA into a world-class observation handling system. Many important details relevant to the IODA design and architecture surfaced during the time we spent together. We mention here only a few highlights. Although the Human Genome Project is famous for its dealing with big data, the amount of observation data that we are dealing with is roughly an order of magnitude larger. When you consider the model data that are also involved in DA flows, that difference becomes three times greater. The manner in which observation data is collected, organized and stored in files is incredibly diverse, and yet there appears to be real promise in creating a common method, or a very small set of methods, for organizing and storing those data within IODA. Attaining a common method of organizing and storing the vast amounts of observation data will go a long way in fostering the collaboration, exchange, and sharing of scientific results.
Dr. Yannick Trémolet, JEDI Project Lead, presenting at the 2019 IODA Workshop in Monterey, CA from February 11 - 13th.
The persistence and resourcefulness of the workshop participants were put to the test Wednesday morning when a wind storm knocked out the power and internet at the meeting site and surrounding neighborhood. Within an hour, communication between on-site and off-site participants was restored using a cell-phone and bluetooth speaker, with local participants clustered around laptops, allowing the day’s presentations and discussion session to be completed on time. This is a further testament to the collaborative spirit and can-do attitude of the partner organizations.
Monterey proved to be a wonderful venue for the workshop. Many fabulous restaurants and sights to see were at our disposal, including the world famous Monterey Aquarium and the beautiful sights along the Pacific Coast Highway.
Thanks are due to the organizers of the workshop, Thomas Auligne (JCSDA), Nancy Baker (NRL) and Yannick Trémolet (JCSDA); moderators of the discussion session, Daryl Kleist (NOAA), Ron Gelaro (NASA) and Nancy Baker (NRL) and all of the participants who contributed presentations and commentary. Special thanks go to Katherine Shanahan (JCSDA) who handled all of the necessary logistics for the workshop (including a last-minute venue change required as a consequence of the government shutdown), as well as Ben Ruston (NRL) and Elizabeth Satterfield (NRL) who organized the social activities that built the commaradie of the group. The organizers and the JEDI team are grateful to all participants for making the 2019 IODA Workshop remarkably successful, productive, and enjoyable.