JCSDA's Q1 Review Showcases Exciting Milestones

On July 18 the JCSDA team met with our partners at NASA, NOAA, US Navy, US Air Force, and the UK Met Office to celebrate the last quarter of accomplishments and discuss goals for the upcoming quarter and year. Q1 milestones included beginning our space weather program and the establishment of a new model interface team.

  • Obs

    • Lots of in kind work continues putting in the last mile employing the genericity of JEDI to construct configuration for the center specific implementation goals

    • Rate of new feature development is beginning to slow, this speaks to the maturity of system and figuring out the correct combination of existing features to create desired outcome.

    • JCSDA OBS team is actively testing FV-3 and MPAS interfaces in JEDI, also in-kinds are testing the other interfaces such as the UK-Met Office LFRic, NRL with the NEPTUNE and NASA with GEOS.  JCSDA COMPO team is actively testing with GEOS as well.

    • Would like to note the relatively new addition of a NASA daily regression test, which has provided fantastic instant feedback – allowing JEDI to react and correct issues immediately

    • Partners all at a high level of proficiency with UFO system

    • JCSDA INFRA and OBS team has developed an automated ingest (IODA) and storage (R2D2) that can be used near real time or for retrospective purposes

    • Space weather is a new area of exploration, this is being led primarily by NRL with JCSDA coordinating

    • This past quarter focused on reduction of technical debt, this included consolidation of some converters for snow cover and soil moisture and removal of deprecated crests and files, and deprecated scripts

    • There has been a lot of progress to complete end-to-end testing including work to produce summary of the forecast scores, specifically bias and rmse with respect to self, a control analysis, or an external analysis such as ERA-5.

    • JCSDA has also begun testing the use of METplus, which is also possible to use with Skylab demonstration runs

    • Working to find a public archive (feed) for ionosondes data, this would be excellent to allow other partners to experiment if they do not have this data easily accessible.  Also opens the potential for external such as University partners, to develop and contribute back to JEDI.

    • NOAA EMC: 

      • initial JEDI implementation is there and working

      • Python interface extended and enhanced

      • Started work on prototyping a database to allow direct access to a data lake based on the NCEP data tanks (TANK DB).

    • UFO status:

      • Added GOES ABI reflectances

      • Space-borne RADAR collaboration with Dr. Isaac Moradi at NASA

      • CRTM v3.1 implemented as operator for radiances

      • Improved UFO code and additional diagnostics/tools

      • Working with the UK Met Office on obs space dataframe implementation

      • Extending OOPS QC flags through UFO, CRTM

      • Ongoing development with partners of surface winds operator

      • In-kinds and OBS team working closely and testing completed ground radar and lightning operators with OU CAPS and NOAA 

      • OBS team working with UKMO on a multi-purpose radar operator

    • GNSS ARO

      • Uses aircraft retrieved bending angle from GNSS-RO receivers, instead of the traditional spaceborne GNSS-RO measurements.  Largely from the Atmospheric River field campaigns.

        • Collaboration with Scripps and UC San Diego, has involved UCAR MMM as well

        • The is a IODA converter for the campaign measurements

        • Continue to work with in-kinds to demonstrate this capability

    • NOAA CWDP Impact assessment of GNSS-R OSW

      • Scores are calculated using JEDI SkyLab capabilities

      • Verification was expanded from exclusively self-analysis, to allow comparison against common control or an external analysis (e.g. ERA-5)

      • Added windborne balloonsondes in collaboration with USAF, doing assimilation impact testing

      • Working with Oklahoma University inkinds on radar and lightning operators, which have been tested in SkyLab; assimilated using JEDI-LETKF

      • Added new tools for regional application monitoring, including ability to customize region

    • NOAA EMC/NCEP progress: 

      • Created the JEDI configuration builder that links their systems and JEDI-usable files

      • Polar AMVs and LEO-GEO AMVs now validated. Geostationary AMVs, scatwind, ozone and most radiances are successfully validated.

    • NRL progress:

      • NRL is leading a collaborative effort to introduce capability for Space Weather assimilation (more details on this in the next section)

      • Implementation of scatterometer winds in FALCON was completed

      • Currently working on OMPS Nadir Profiling Ozone observations

    • GMAO progress:

      • Contributed to bug fixes in the JEDI/UFO linear operators for radiosonde and ozone observations

      • Investigated the tropopause calculation in JEDI, and used this to find in GOES-GSI an incorrect divergence/vorticity calculation

      • Updated SWELL observational yaml configurations and Python scripts for update in JEDI conventions related to bias correction

      • Investigated the inconsistency in “geoval”  from OMI/AURA ozone between GSI and JEDI, which was found due to the different order of vertical pressure levels

      • Diagnosed a CO2 unit inconsistency in the background state provided that was corrected in configuration. This was found due to anomalous increments in IR observations seen in the JEDI-GOES regression testing.

    • UK Met Office progress:

      • IASI, CrIS, AIRS are in and performing in a comparable manner to current Met Office operational DA system

      • Identified a couple of bugs in the RTTOV interface associated with RTTOV-Scatt in the LinearObsOperator which have now been fixed and this is working as expected

      • Implemented a full radar Doppler wind processing chain in our limited area (UKV) model. This closely matches the existing (OPS) system

    • Plans for next quarter:

      • Continue validating, testing, and implementing forward operators and filtering functions to enhance UFO versatility and adaptability

      • UFO-CRTM development/improvement for UV and visible

      • Skylab demonstrator for benchmarking, higher order regression testing and to perform impact studies

      • GDAS 4D-FGAT proxy app demonstration with configuration provided by NOAA

      • Development for space weather in UFO

  • Space Weather

    • New initiative: JEDI for space weather

    • Goal: develop JEDI capabilities for assimilation of thermosphere and ionosphere observations to enhance space weather prediction capabilities

    • In Q1/Q2, will work on data ingest, then in Q3/Q4 on forward operator development and extension

    • Good progress has already been made, with two PRs created based on sample data provided by NRL and a PyIRI-JEDI repository was established on JCSDA JEDI github

    • Working with JEDI core team on adding PyIRI-JEDI to JEDI-bundle

    • Generating GeoVal files and testing the vertical interpolation operator for computing H(x) for an ionosonde electron density profile 

    • A kick-off meeting for the project will be held on August 28-29, with virtual and in-person attendance options 

      • The meeting will gather key stakeholders involved in developing advanced data assimilation systems within the JEDI framework for ionosphere-thermosphere-mesosphere (ITM) space weather forecasting models, and assess the roles and contributions of each participant and explore how resources can be effectively shared to achieve common objectives

  • CRTM

    • The latest release, v3.1, added active radar support, cmake support for build/compile (no ecbuild required), experimental visible radiance reflectance output, and enhanced netCDF support

    • Working on eliminating all binary files and switching to netCDF

    • Adding capability to do visible reflectance, simulate ABI reflectance, and be able to directly assimilate visible radiance

    • Bugfixes under way for aircraft pressure levels jacobean and cloud fraction edge case of tiny cloud fraction

    • V3.1.1 will be a bugfix release (released August 12, 2024)

    • V3.2 is currently being designed and will contain substantial updates and new features

      • Stretch goal: generic method for surface objects

    • Offline package for generating CRTM optical profiles is ready and under testing. This package supports CRTM calculations with the abstract optical interface

    • Continue collaboration with NCAR colleagues on GEMS/WRF-Chem/MPAS AOD assimilation

    • Fixed effective radius selection in CRTM/UFO calculations with CMAQ LUT

    • NOAA-STAR developed CRTM with embedded radar equation (Dr. Isaac Moradi) 

    • NOAA-STAR continues to work on coefficient generation

      • Generated coefficients for INSAT-3DS (Indian National Satellite - 3D second repeat), a geostationary instrument with a 19-channel sounder and a 6-channel imager

      • The CRTM SpecCoeff format was updated to include a new variable: the polarization angle

    • NASA GMAO progress:

      • Improving the UFO/CRTM interface for the assimilation of radar observations within JEDI including enhancement for the handling of the elevation dimension in radar observations and also the missing values where there is cloud in the background profiles provided by the model

      • Developing observation formats for both CloudSat CPR and GPM DPR compatible with the JEDI IODA 

      • Implementation of the CRTM radar simulator within SkyLab 

      • Preparing sample observations for the SkyLab 2022 reference period

      • Preparing YAML files for the assimilation of CloudSat CPR and GPM DPR

      • Working on multiple scattering impact on radar observations

    • Progress on CRTM AI project, which is joint work between Lucas Howard (CU) and Greg Thompson (JCSDA)

      • Lucas Howard, CU Boulder, has been working on using CRTM as training for a neural network, with help from Greg Thompson

      • 3 hidden layers x 512 nodes per layer

      • Some tuning to arrive at this architecture using earlier datasets

      • ~1.1 million trainable parameters

      • Input: All CRTM profile, surface, and meta input variables

      • Output: Predicted CRTM ABI brightness temperature for channels 7-16, predicted error (NN-CRTM) standard deviation by channel

      • Cost function (to be minimized): Continuous rank probability score (CRPS), penalizes inaccuracy and imprecision

    • Upcoming work:

      • Standardize and improve the NetCDF interface for all coefficient LUTs

      • Add UV/Visible/IR reflectance models for snow/ice

      • Aerosol coefficient generation package

      • Genericization of surface and optical property interfaces, separation of concerns (exploratory)

      • Initial pivot toward AI front-end development for CRTM (CRTM v4.x).   Coordinated effort with JCSDA partners, CU-Boulder

  • SOCA

    • AOP24 overall priorities: 

      • operational SOCA ready for EMC and GMAO (global ¼ degree hybrid LETKF 3DEnVar marine DA, special focus on ice DA developments)

      • Scientific advancements: coupled ocean/atmosphere DA, 4D ocean DA methods

    • Q1 accomplishments included:

      • SOCA now properly uses SABER

      • Diffusion operator made generic and moved to OOPS/SABER

      • Now using SABER block for localization and correlation in SOCA

      • Maintenance of SOCA code, including removing a lot of fortran and refactoring the remaining fortran into C++

    • Dr. Kriti Bhargava has been adding the UFS marine model to SkyLab

    • Using marine UFS in cycling 3DVar

    • Ocean 4DVar with HTLM

      • Testing with 5 degree increments are working

      • Next quarter will test with a realistic case (¼ deg regional) and compare with existing SOCA DA methods

    • Ocean color (OASIM) is making good progress

      • Added 0.25 deg NOBM biochemistry

      • ocean color hofx added to marine SKYLAB

      • Goal for next quarter is coupled hofx

    • SOCA contributions and usage at NASA GMAO:

      • GEOS is cycling with CICE6 and sea ice DA

      • GMAO will be collaborating with EMC and JCSDA on sea ice DA testing, development, and tuning

    • SOCA contributions and usage at NOAA EMC:

      • GFSv17 prototype evaluation

      • began implementing LETKF in the EMC workflow

      • Added RTOFS insitu T/S observations and additional sea-ice concentration observations

  • Operations

    • With AOP24, adopting a requirements-based methodology

      • AOP24 will consist of 3 main documents supported by reference materials

      • AOP24 is now with the MOB, and should be approved in the next few weeks

      • A new events page with upcoming conferences and workshops in forecasting and data assimilation has been added to the website here

  • COMPO

    • Accomplished this quarter:

      • Cycling with GEOS: Cycling is now possible with the GEOS IAU replay mode using GEOS-FP

      • Cost ratios: both an and fc are at c90. Next the team will increase to 32 members and start scientific validation and tuning of the system

    • TEMPO

      • Ran experiment with TEMPO NO2 data

      • Able to reduce biases in hotspots

      • Designing experiment for August to do scientific validation: this experiment will have more observations included, and will run the GEOS cf model at multiple resolutions

    • Hybrid TLM testing showed a reduction in error compared to simple TLM; reduced linear error of chem variables and showed a better fit to observations for Tropomi NO2

    • L1 albedo aerosol DA:

      • Added IODA for VIIRS NO2 albedo data

      • Was able to see some plumes not captured in previous work

      • TOA albedo from GEFS-Aerosols shows significant underestimation at 550nm. More tests are undergoing

      • JEDI-METplus workflow has been updated for running on Derecho

    • NOAA OAR/ JCSDA achievements this quarter: 

      • Developed Vader transforms to assimilate PM25 in the CMAQ model, an excellent example of collaboration and joint success

      • now investigating the use of Vader to map CMAQ aerosol to NASA GOCART LUT

      • Potential ML/AI aid in the partitioning and AOD calculation (Jerome) - i.e. determine the unknown partitioning coefficients for OC and BC

    • NOAA NCEP/EMC progress:

      • Initial testing of 3DVar global aerosol DA at C384 resolution

      • Initial test of diffusion correlations for B

      • Work has continued on tuning and evaluating the variance tuning methods for B in FV3-LAM-CMAQ for NO2

  • Infrastructure

    • Applications, data infrastructure, software environment, and cloud resources are all on track

    • EWOK/SkyLab development:

      • On target

      • Added atmospheric composition obs to ingest suite

      • Added ability to ingest data at shorter time windows, demonstrated with GOES data at 10 minute steps

      • Demonstrated the enhanced capability to monitor and produce analyses of atmospheric composition less than 24 hours after official product release with TEMPO NO2 data utilizing the EWOK ingest suite

    • R2D2 is moving from directly interacting with the interface to a REST API, which is better for security, maintenance, and control

      • The openAPI spec for the REST API is in testing

    • The R2D2 team, along with Dr. Tariq Hamzey of NASA GMAO and several other JCSDA JEDI team members, published and presented a peer-reviewed paper at the International Conference on Computational Science in early July

    • Spack-stack progress:

      • On track for spack-stack-1.8.0 release, which is targeted for late Aug/early Sep, 2024

      • Reduced need for multiple library versions

      • Returning to a quarterly cadence for releases

      • Progress on the switch to Intel oneAPI LLVM based compilers

      • Upcoming work: spack-stack-1.8.0 release, continue work on the switch to Intel oneAPI LLVM based compilers, ramp up work on automating periodic build/test cycles of spack-stack on all the supported HPC platforms

    • Cloud infrastructure progress:

      • Allocated new MPAS high-res cluster

      • Full software environment overhaul: Ubuntu 22.04, Parallel Cluster 3.7.1, spack-stack 1.7

      • Allocated and configured cloud projects for UK Met Office GPU optimization

    • Upcoming work for AOP24:

      • Demonstrate continuous observation ingest with temporal slicing

      • Add more observations to ingest workflow

      • Integrate spack-stack test environment with nightly skylab tests

  • Models Interface

    • New team whose scope includes the interface between OOPS/JEDI and models, generic algorithms on model data, and optimizations (memory, CPU, GPU)

    • Highlights this quarter: onboarding Liam Adams, kicking off the GPU offloading project, and ongoing improvements to generic interpolators

    • GPU Offloading project is a joint effort with the UK Met Office

      • Taking some generic code, like b matrix computations, and performing on GPU

      • Portability is a driving goal

      • This quarter, the team will review GPU programming approaches and conduct exploratory tests using Atlas data structures and OpenMP kernels

    • Generic interpolators have two key goals:

      • Scientific: flexibility to share SABER covariances across models

      • Optimization: improve speed and memory of model resolution changes

    • Plans for Q2:

      • prepare JEDI interfaces for space-weather models

      • use generic interpolators in fv3-jedi & mpas-jedi resolution changes

      • offload JEDI algorithms to GPUs

  • Algo

    • Francois Hebert moved to a new role, leading the Models Interface team

    • Major algo themes for AOP24:

      • Generic development in IODA, VADER, SABER, and EWOK/SkyLab

      • Multiscale background error covariances with Skylab-HRDAS

      • Ensemble of Data Assimilation with Skylab-GEOS

      • Continuous Data Assimilation with Skylab-HRDAS

      • Hybrid TLM development and scientific validation

    • IODA updates:

      • UKMO completed implementation of ObsSpace data container class

      • NOAA-EMC completed initial implementation of BUFR reader backend

      • ReduceObsSpace filter action has been added to UFO

      • Completed initial implementation of multiple input file handling (including in Skylab)

    • H(x) with the reduce ObsSpace QC filter showed order of magnitude reduction of memory footprint, and ~70% reduction of runtime!

    • Significant growth in VADER, with 17 new recipes added by inkinds this quarter

    • Most of the work for adding VADER to MPAS-JEDI is done

    • Plans continue moving forward to adopt model variable naming convention in JEDI

      • While planning the code sprint for this project, lots of ways to improve the code and make implementation much easier were found

    • Update to background error covariance training suite to be more robust and flexible 

    • Snow depth DA is now getting reasonable results and runs in less than 3 minutes

      • In final testing for GFSv17

      • Option for different background covariance from SABER blocks

    • Goal for this year: multiscale background error covariance

      • using the spectral analytical filter block for ensemble covariance localization is in progress

    • Another big goal for this year is EDA with SkyLab-GEOS, aiming to emulate the current operational system for the high resolution system

      • Simple variational case is complete

      • Next steps is making it more sophisticated with more obs and better covariances, and do EDA later this year

    • The HTLM is showing good reduction in linearization errors

    • Another theme for this year: continuous DA

      • Introduce new options in variational DA

      • Work is in progress on capability to add obs added since last outer loop in the next outer loop

    • Plans for Q2:

      • Generic development in:

        • IODA: ObsDataFrame integration for more flexible and efficient data storage

        • VADER: improvements to linear variable change for more flexibility and use in SABER

      • Multiscale B: more extensive testing of spectral analytical filter with mpas-jedi (and fv3-jedi?); initial tests with multiscale covariances with mpas-jedi.

      • Skylab-GEOS: 3D-Var with most observations, hybrid B (GSI + BUMP)

      • Continuous DA: ability to add newly arrived observations

Congratulations to the entire JCSDA staff and all of our partners and in-kinds for these big steps forward! Keep an eye out for upcoming developments with space weather prediction, visual reflectance DA, and coupled DA!

Photo by W on Unsplash