On April 18 the JCSDA team met with our partners at NASA, NOAA, US Navy, US Air Force, and the UK Met Office to celebrate the last quarter of accomplishments and discuss goals for the upcoming quarter and AOP year. Q4 accomplishments included several important bugfixes, new sensors, and several experiments added to SkyLab v8!
The Observations team started the morning off, with quarterly accomplishments including:
Non-cycling hybrid 3DEnVar demonstrated
Weather-variable-grid demonstration with MPAS, with cycling 3DEnVar
Now running CRTM 3.1 for relevant instruments, including AOD retrieval
Balloonsonde converter updated
ncDiag converter update, this combines background and analysis ncDiag files into single feedback file, update for GNSS-RO completed
GNSS-R winds from multiple providers have been ingested and are being evaluated
GNSS-R ocean surface wind in development with UK Met Office
Finalizing VarBC updates to standardize to IODA conventions
SMAP soil moisture converter has been updated to current conventions and monitoring included in SkyLab v8
TROPICS proxy data update to TROPICS converter, readiness for future data release
EMC, added a script backend to IODA which lets them run a python script to generate directly in UFO an obsgroup object
UFO status: added active assimilation of GOES ABI reflectance, this is in addition to already demonstrated brightness temperatures. Both are demonstrated in Skylab v8 with CRTM v3.1. GOES sensor has a lot of untapped potential remaining;
With OU-CAPS working to integrate into UFO a GOES Geostationary Lightning Mapper (GLM) Flash Extent Density (FED) Observation Operator
EUMETSAT ROM-SAF project with GNSS-RO continues to abstract and insert multiple super refractivity checks, methods derived from the different partners;
Collaborating with UCAR for dynamic error estimation for GNSS-RO utilizing the local spectral width (LSW)
NOAA EMC continues work on UFO acceptance: Geostationary AMVs, scatwind, ozone and most radiances are successfully validated, issues with ozone operator and geostationary data were solved
NRL continues advanced integration testing including cycling with updating VarBC coefficients, these are being comparing NEPTUNE DA with NAVGEM
UK Met Office has completed the observations process porting to the JEDI-based one for the global model and is shifting focus to the regional model.
A JEDI-based global observation processing is planned to be operational in their next update, mid next year.
Next quarter plans: formalize new AOP, UFO/CRTM development and improvement for UV and visible spectrum, complete VarBC for and conventional data for aircraft, develop framework to perform impact studies, and developments for space weather capabilities in UFO
CRTM followed Obs, with updates on:
No more separation of code base with JEDI partners, facilitating collaborative efforts
Strengthened radar support ongoing
Added support for visible radiance reflectance
Cmake support added for build/compile, ecbuild no longer required
Contributions from GMAO:
New cloud coefficient for frequencies ranging up to 800 GHz
Modifying radar simulator to work for different zenith angles
Generating microphysics-consistent cloud coefficient (in progress)
CRTM generic optical interface ready and being tested
Users can import cloud/aerosol optical properties for CRTM forward calculations without CRTM cloud/aerosol LUTs
Updated CRTM GOCART-GEOS5 aerosol coefficient
Enabled all optional CRTM aerosol LUTs for aerosol DA in JEDI
Support package for cloud/aerosol optical profiles calculations under development
Released CRTM v3.1 and accompanying documentation
Contributions from STAR:
Generated CRTM coefficients for microwave imagery on WFS-M1, infrared sounder on meteosat third generation, and a research version of CRTM coefficients for NOAA-21 OMPS sensors
MTGRS development coming along well
Goals for next quarter:
V3.1.1 development focusing on full build integration with partner requirements
Cloud tables containing oriented hydrometeors for polarization
Updates to MW/IR land service emissivity LUTs
Updates to sea surface emissivity fast modeling in IR/MW
Melting layer model integration
Ongoing conversion from binary to netCDF format for all coefficient types, defaulting to netCDF distribution with binary files available on demand
Community engagement through CRTM user/developer workshop, code sprints, seminars, and STEM outreach in middle and high school programs
Next up was SOCA:
Added forecast experiment in SkyLab using MOM6-CICE6-DATM version of ufs
Getting close to integrating the forecast experiment with cycling 3DVar experiments
Lots of cleaning up of tech debt accomplished this quarter
Refactored parametric ocean standard deviation in preparation for moving to SABER
Also refactored diffusion correlation/localization operator
Diffusion operator is now model-agnostic
Diffusion operator is being used as a generic smoother, which has been useful
Approximately 15% of existing SOCA code was deleted this quarter with no impact to capabilities
Lots of progress made on adding OASIM h(x) to SkyLab
New converters for in situ obs will be ready soon
Infra added Torch to spack-stack, enabling compilation with AI-based operators
EMC and NASA have tested the diffusion correlation operator successfully
Major focus areas for AOP24 include cleanup and generalization of SOCA code, sea-ice DA testing and improvements, and coupled radiance DA
Director's office/operations:
Transferring epics and features to the new AOP is in progress
Contributions from partner agencies increased this quarter
Canceled the Science Workshop to focus on implementation and operational systems
AOP24 is being made more requirements-based
The news page was the 10th-most viewed on the website this quarter
Expected drop in LinkedIn views and engagement due to fewer posts
The R2D2 paper for ICCS was accepted
Atmospheric COMPOsition:
For SkyLab v8, added:
4DEnVar with GEOS at C90 resolution with 32 ensemble members
Obs: radiosondes, satwinds, AMSU_A, MHS, VIIRS, MODIS
4DEnVar trace gas with a 1 hour subwindow and c90 backgrounds; the ability to do 4D analysis is very important for TEMPO and emission data assimilation
CRTM AOD cleanup achieved, making the CRTM AOD operator more flexible and capable of loading a lookup table for user-defined aerosol optical properties
Completed preliminary demonstration of JEDI TEMPO, awaiting public data release
Added GEOS 4DVar trace gas experiment and aerosol experiment
Continuing HTLM work with the Algo team to design a trace-gas experiment for scientific evaluation
Working on cycling with GEOS-CF with JEDI
Developed VIIRS N20 data converter for L1B reflectance products
Collaborated with GSL on VADER variable transforms for computing PM2.5 from the CMAQ aerosol scheme
Tuned B-matrix localization for high-resolution TEMPO experiments
NOAA EMC work:
Helped consolidate and refactor FV3-JEDI and UFO tests related to global AOD DA including incorporating VarBC
built a "variance partition" scheme which takes an ensemble of neighboring grid points to compute error standard deviations at each analysis time. This results in a time dependent uncertainty to potentially replace the static B matrix currently used in the aerosol 3dVar DA. It is based on the methodology under development in SOCA at EMC.
continue to work on the Integration of the JEDI-based DA capability for trace gas (NO2) into the production version 'aqm_dev' of the regional workflow
The Algorithms team provided highlights including:
Quarter highlights:
Added FOV averaging to UFO CRTM operator, which improves matching to obs and GSI and supports masked interpolation
Improved robustness of generic algorithms, particularly the halo point fix (which also fixed several GSI-JEDI issues) and an improved method of specifying variables for coupled applications
Significant progress towards model naming convention adoption, which is crucial for maintenance; this project is a collaboration with our partner agencies, and the UK Met Office in particular has made many contributions this quarter
Finalized full support of hybrid 4DEnVar covariances
In SkyLab, added: HTLM experiments with GFS, GEOS forecast, more efficient cycling, B matrix training workflow
Preliminary experiments for snow 2DVar
Added support for hybrid covariances in 4DEnVar in JEDI with GSI, and now 4D is supported for the hybrid GSI covariance SABER block (joint work with Rocardo Todling at GMAO)
Testing multi-resolution HTLM with a residual formulation enabling higher resolution updates; running high-res experiments comparing this with 4DEnVar and standard 4DVar with fv3-tlm
Working on the GEOS forecast in sky-lab, which can be built with spack-stack on Discover; next is building and running on other platforms and adding a variational run
Preliminary AOP24 plans:
Generic code and model interface maintenance and generalization
Generic code optimization on CPU and GPU
JEDI interface with AI models
Testing and scientific evaluation of 4DEnVar, 4DVar, LETKF, EDA
Multiscale background error covariances
Continuous DA
Finishing up the review was the Infrastructure team
Software environment, testing and CI/CD, R2D2 development, and cloud and github administration are on target; IODA and EWOK/SkyLab development are behind
Released spack-stack 1.7.0 with NOAA EMC and EPIC
Support for JEDI with FV3, UFS, MPAS, UM, NEPTUNE, GEOS
Support for UFS applications
Support for GEOS-GCM and NASA’s SWELL environment
Support for cylc-8
UKMO and NOAA have contributed a lot to memory/runtime performance profiling efforts for IODA
Lots of progress on ObsSpace Memory Data Container
R2D2 has updated reference documentation, better storage and retrieval of SkyLab experiment configurations, and automation of experiment cleaning on all platforms
AWS parallelcluster R&D (Research & Development) system was shut down, will be launched again in AOP24
Solved some issues with NOAA ParallelWorks, which will be ready for SkyLab v9
Added several more observations to EWOK ingest suite
EWOK/SkyLab now successfully ingests, converts to different resolutions, and runs forecasts for MPAS deterministic and ensemble backgrounds
The EWOK/SkyLab observation/model ingest suite can now be run on Orion and Hercules
The new CI system now covers all repositories in jedi-bundle, and the caching strategy and handling of Fortran module files has been improved. Flaky/failing unit test issues were resolved
Github management has been automated for 40 repositories in the internal JEDI github
Preliminary plans for AOP24:
Continued support of spack-stack for JEDI-SkyLab and partner models
Expand and support new CI system
Reader/writer performance improvements for IODA, ObsSpace dataframe implementation
Roll out R2D2 REST API implementation
Add support for cylc-8 in EWOK/SkyLab
Improved interface for visualizing diagnostics in EWOK/SkyLab
Congratulations to the entire JCSDA staff and all of our partners and in-kinds for these big steps forward! Keep an eye out for upcoming developments with the R2D2 REST API implementation, TEMPO data, and snow 2DVar!
Photo by Bill Jelen on Unsplash