On July 18 the JCSDA team met with our partners at NASA, NOAA, US Navy, US Air Force, and the UK Met Office to celebrate the last quarter of accomplishments and discuss goals for the upcoming quarter and year. Q1 milestones included beginning our space weather program and the establishment of a new model interface team.
Obs
Lots of in kind work continues putting in the last mile employing the genericity of JEDI to construct configuration for the center specific implementation goals
Rate of new feature development is beginning to slow, this speaks to the maturity of system and figuring out the correct combination of existing features to create desired outcome.
JCSDA OBS team is actively testing FV-3 and MPAS interfaces in JEDI, also in-kinds are testing the other interfaces such as the UK-Met Office LFRic, NRL with the NEPTUNE and NASA with GEOS. JCSDA COMPO team is actively testing with GEOS as well.
Would like to note the relatively new addition of a NASA daily regression test, which has provided fantastic instant feedback – allowing JEDI to react and correct issues immediately
Partners all at a high level of proficiency with UFO system
JCSDA INFRA and OBS team has developed an automated ingest (IODA) and storage (R2D2) that can be used near real time or for retrospective purposes
Space weather is a new area of exploration, this is being led primarily by NRL with JCSDA coordinating
This past quarter focused on reduction of technical debt, this included consolidation of some converters for snow cover and soil moisture and removal of deprecated crests and files, and deprecated scripts
There has been a lot of progress to complete end-to-end testing including work to produce summary of the forecast scores, specifically bias and rmse with respect to self, a control analysis, or an external analysis such as ERA-5.
JCSDA has also begun testing the use of METplus, which is also possible to use with Skylab demonstration runs
Working to find a public archive (feed) for ionosondes data, this would be excellent to allow other partners to experiment if they do not have this data easily accessible. Also opens the potential for external such as University partners, to develop and contribute back to JEDI.
NOAA EMC:
initial JEDI implementation is there and working
Python interface extended and enhanced
Started work on prototyping a database to allow direct access to a data lake based on the NCEP data tanks (TANK DB).
UFO status:
Added GOES ABI reflectances
Space-borne RADAR collaboration with Dr. Isaac Moradi at NASA
CRTM v3.1 implemented as operator for radiances
Improved UFO code and additional diagnostics/tools
Working with the UK Met Office on obs space dataframe implementation
Extending OOPS QC flags through UFO, CRTM
Ongoing development with partners of surface winds operator
In-kinds and OBS team working closely and testing completed ground radar and lightning operators with OU CAPS and NOAA
OBS team working with UKMO on a multi-purpose radar operator
GNSS ARO
Uses aircraft retrieved bending angle from GNSS-RO receivers, instead of the traditional spaceborne GNSS-RO measurements. Largely from the Atmospheric River field campaigns.
Collaboration with Scripps and UC San Diego, has involved UCAR MMM as well
The is a IODA converter for the campaign measurements
Continue to work with in-kinds to demonstrate this capability
NOAA CWDP Impact assessment of GNSS-R OSW
Scores are calculated using JEDI SkyLab capabilities
Verification was expanded from exclusively self-analysis, to allow comparison against common control or an external analysis (e.g. ERA-5)
Added windborne balloonsondes in collaboration with USAF, doing assimilation impact testing
Working with Oklahoma University inkinds on radar and lightning operators, which have been tested in SkyLab; assimilated using JEDI-LETKF
Added new tools for regional application monitoring, including ability to customize region
NOAA EMC/NCEP progress:
Created the JEDI configuration builder that links their systems and JEDI-usable files
Polar AMVs and LEO-GEO AMVs now validated. Geostationary AMVs, scatwind, ozone and most radiances are successfully validated.
NRL progress:
NRL is leading a collaborative effort to introduce capability for Space Weather assimilation (more details on this in the next section)
Implementation of scatterometer winds in FALCON was completed
Currently working on OMPS Nadir Profiling Ozone observations
GMAO progress:
Contributed to bug fixes in the JEDI/UFO linear operators for radiosonde and ozone observations
Investigated the tropopause calculation in JEDI, and used this to find in GOES-GSI an incorrect divergence/vorticity calculation
Updated SWELL observational yaml configurations and Python scripts for update in JEDI conventions related to bias correction
Investigated the inconsistency in “geoval” from OMI/AURA ozone between GSI and JEDI, which was found due to the different order of vertical pressure levels
Diagnosed a CO2 unit inconsistency in the background state provided that was corrected in configuration. This was found due to anomalous increments in IR observations seen in the JEDI-GOES regression testing.
UK Met Office progress:
IASI, CrIS, AIRS are in and performing in a comparable manner to current Met Office operational DA system
Identified a couple of bugs in the RTTOV interface associated with RTTOV-Scatt in the LinearObsOperator which have now been fixed and this is working as expected
Implemented a full radar Doppler wind processing chain in our limited area (UKV) model. This closely matches the existing (OPS) system
Plans for next quarter:
Continue validating, testing, and implementing forward operators and filtering functions to enhance UFO versatility and adaptability
UFO-CRTM development/improvement for UV and visible
Skylab demonstrator for benchmarking, higher order regression testing and to perform impact studies
GDAS 4D-FGAT proxy app demonstration with configuration provided by NOAA
Development for space weather in UFO
Space Weather
New initiative: JEDI for space weather
Goal: develop JEDI capabilities for assimilation of thermosphere and ionosphere observations to enhance space weather prediction capabilities
In Q1/Q2, will work on data ingest, then in Q3/Q4 on forward operator development and extension
Good progress has already been made, with two PRs created based on sample data provided by NRL and a PyIRI-JEDI repository was established on JCSDA JEDI github
Working with JEDI core team on adding PyIRI-JEDI to JEDI-bundle
Generating GeoVal files and testing the vertical interpolation operator for computing H(x) for an ionosonde electron density profile
A kick-off meeting for the project will be held on August 28-29, with virtual and in-person attendance options
The meeting will gather key stakeholders involved in developing advanced data assimilation systems within the JEDI framework for ionosphere-thermosphere-mesosphere (ITM) space weather forecasting models, and assess the roles and contributions of each participant and explore how resources can be effectively shared to achieve common objectives
CRTM
The latest release, v3.1, added active radar support, cmake support for build/compile (no ecbuild required), experimental visible radiance reflectance output, and enhanced netCDF support
Working on eliminating all binary files and switching to netCDF
Adding capability to do visible reflectance, simulate ABI reflectance, and be able to directly assimilate visible radiance
Bugfixes under way for aircraft pressure levels jacobean and cloud fraction edge case of tiny cloud fraction
V3.1.1 will be a bugfix release (released August 12, 2024)
V3.2 is currently being designed and will contain substantial updates and new features
Stretch goal: generic method for surface objects
Offline package for generating CRTM optical profiles is ready and under testing. This package supports CRTM calculations with the abstract optical interface
Continue collaboration with NCAR colleagues on GEMS/WRF-Chem/MPAS AOD assimilation
Fixed effective radius selection in CRTM/UFO calculations with CMAQ LUT
NOAA-STAR developed CRTM with embedded radar equation (Dr. Isaac Moradi)
NOAA-STAR continues to work on coefficient generation
Generated coefficients for INSAT-3DS (Indian National Satellite - 3D second repeat), a geostationary instrument with a 19-channel sounder and a 6-channel imager
The CRTM SpecCoeff format was updated to include a new variable: the polarization angle
NASA GMAO progress:
Improving the UFO/CRTM interface for the assimilation of radar observations within JEDI including enhancement for the handling of the elevation dimension in radar observations and also the missing values where there is cloud in the background profiles provided by the model
Developing observation formats for both CloudSat CPR and GPM DPR compatible with the JEDI IODA
Implementation of the CRTM radar simulator within SkyLab
Preparing sample observations for the SkyLab 2022 reference period
Preparing YAML files for the assimilation of CloudSat CPR and GPM DPR
Working on multiple scattering impact on radar observations
Progress on CRTM AI project, which is joint work between Lucas Howard (CU) and Greg Thompson (JCSDA)
Lucas Howard, CU Boulder, has been working on using CRTM as training for a neural network, with help from Greg Thompson
3 hidden layers x 512 nodes per layer
Some tuning to arrive at this architecture using earlier datasets
~1.1 million trainable parameters
Input: All CRTM profile, surface, and meta input variables
Output: Predicted CRTM ABI brightness temperature for channels 7-16, predicted error (NN-CRTM) standard deviation by channel
Cost function (to be minimized): Continuous rank probability score (CRPS), penalizes inaccuracy and imprecision
Upcoming work:
Standardize and improve the NetCDF interface for all coefficient LUTs
Add UV/Visible/IR reflectance models for snow/ice
Aerosol coefficient generation package
Genericization of surface and optical property interfaces, separation of concerns (exploratory)
Initial pivot toward AI front-end development for CRTM (CRTM v4.x). Coordinated effort with JCSDA partners, CU-Boulder
SOCA
AOP24 overall priorities:
operational SOCA ready for EMC and GMAO (global ¼ degree hybrid LETKF 3DEnVar marine DA, special focus on ice DA developments)
Scientific advancements: coupled ocean/atmosphere DA, 4D ocean DA methods
Q1 accomplishments included:
SOCA now properly uses SABER
Diffusion operator made generic and moved to OOPS/SABER
Now using SABER block for localization and correlation in SOCA
Maintenance of SOCA code, including removing a lot of fortran and refactoring the remaining fortran into C++
Dr. Kriti Bhargava has been adding the UFS marine model to SkyLab
Using marine UFS in cycling 3DVar
Ocean 4DVar with HTLM
Testing with 5 degree increments are working
Next quarter will test with a realistic case (¼ deg regional) and compare with existing SOCA DA methods
Ocean color (OASIM) is making good progress
Added 0.25 deg NOBM biochemistry
ocean color hofx added to marine SKYLAB
Goal for next quarter is coupled hofx
SOCA contributions and usage at NASA GMAO:
GEOS is cycling with CICE6 and sea ice DA
GMAO will be collaborating with EMC and JCSDA on sea ice DA testing, development, and tuning
SOCA contributions and usage at NOAA EMC:
GFSv17 prototype evaluation
began implementing LETKF in the EMC workflow
Added RTOFS insitu T/S observations and additional sea-ice concentration observations
Operations
With AOP24, adopting a requirements-based methodology
AOP24 will consist of 3 main documents supported by reference materials
AOP24 is now with the MOB, and should be approved in the next few weeks
A new events page with upcoming conferences and workshops in forecasting and data assimilation has been added to the website here
COMPO
Accomplished this quarter:
Cycling with GEOS: Cycling is now possible with the GEOS IAU replay mode using GEOS-FP
Cost ratios: both an and fc are at c90. Next the team will increase to 32 members and start scientific validation and tuning of the system
TEMPO
Ran experiment with TEMPO NO2 data
Able to reduce biases in hotspots
Designing experiment for August to do scientific validation: this experiment will have more observations included, and will run the GEOS cf model at multiple resolutions
Hybrid TLM testing showed a reduction in error compared to simple TLM; reduced linear error of chem variables and showed a better fit to observations for Tropomi NO2
L1 albedo aerosol DA:
Added IODA for VIIRS NO2 albedo data
Was able to see some plumes not captured in previous work
TOA albedo from GEFS-Aerosols shows significant underestimation at 550nm. More tests are undergoing
JEDI-METplus workflow has been updated for running on Derecho
NOAA OAR/ JCSDA achievements this quarter:
Developed Vader transforms to assimilate PM25 in the CMAQ model, an excellent example of collaboration and joint success
now investigating the use of Vader to map CMAQ aerosol to NASA GOCART LUT
Potential ML/AI aid in the partitioning and AOD calculation (Jerome) - i.e. determine the unknown partitioning coefficients for OC and BC
NOAA NCEP/EMC progress:
Initial testing of 3DVar global aerosol DA at C384 resolution
Initial test of diffusion correlations for B
Work has continued on tuning and evaluating the variance tuning methods for B in FV3-LAM-CMAQ for NO2
Infrastructure
Applications, data infrastructure, software environment, and cloud resources are all on track
EWOK/SkyLab development:
On target
Added atmospheric composition obs to ingest suite
Added ability to ingest data at shorter time windows, demonstrated with GOES data at 10 minute steps
Demonstrated the enhanced capability to monitor and produce analyses of atmospheric composition less than 24 hours after official product release with TEMPO NO2 data utilizing the EWOK ingest suite
R2D2 is moving from directly interacting with the interface to a REST API, which is better for security, maintenance, and control
The openAPI spec for the REST API is in testing
The R2D2 team, along with Dr. Tariq Hamzey of NASA GMAO and several other JCSDA JEDI team members, published and presented a peer-reviewed paper at the International Conference on Computational Science in early July
Spack-stack progress:
On track for spack-stack-1.8.0 release, which is targeted for late Aug/early Sep, 2024
Reduced need for multiple library versions
Returning to a quarterly cadence for releases
Progress on the switch to Intel oneAPI LLVM based compilers
Upcoming work: spack-stack-1.8.0 release, continue work on the switch to Intel oneAPI LLVM based compilers, ramp up work on automating periodic build/test cycles of spack-stack on all the supported HPC platforms
Cloud infrastructure progress:
Allocated new MPAS high-res cluster
Full software environment overhaul: Ubuntu 22.04, Parallel Cluster 3.7.1, spack-stack 1.7
Allocated and configured cloud projects for UK Met Office GPU optimization
Upcoming work for AOP24:
Demonstrate continuous observation ingest with temporal slicing
Add more observations to ingest workflow
Integrate spack-stack test environment with nightly skylab tests
Models Interface
New team whose scope includes the interface between OOPS/JEDI and models, generic algorithms on model data, and optimizations (memory, CPU, GPU)
Highlights this quarter: onboarding Liam Adams, kicking off the GPU offloading project, and ongoing improvements to generic interpolators
GPU Offloading project is a joint effort with the UK Met Office
Taking some generic code, like b matrix computations, and performing on GPU
Portability is a driving goal
This quarter, the team will review GPU programming approaches and conduct exploratory tests using Atlas data structures and OpenMP kernels
Generic interpolators have two key goals:
Scientific: flexibility to share SABER covariances across models
Optimization: improve speed and memory of model resolution changes
Plans for Q2:
prepare JEDI interfaces for space-weather models
use generic interpolators in fv3-jedi & mpas-jedi resolution changes
offload JEDI algorithms to GPUs
Algo
Francois Hebert moved to a new role, leading the Models Interface team
Major algo themes for AOP24:
Generic development in IODA, VADER, SABER, and EWOK/SkyLab
Multiscale background error covariances with Skylab-HRDAS
Ensemble of Data Assimilation with Skylab-GEOS
Continuous Data Assimilation with Skylab-HRDAS
Hybrid TLM development and scientific validation
IODA updates:
UKMO completed implementation of ObsSpace data container class
NOAA-EMC completed initial implementation of BUFR reader backend
ReduceObsSpace filter action has been added to UFO
Completed initial implementation of multiple input file handling (including in Skylab)
H(x) with the reduce ObsSpace QC filter showed order of magnitude reduction of memory footprint, and ~70% reduction of runtime!
Significant growth in VADER, with 17 new recipes added by inkinds this quarter
Most of the work for adding VADER to MPAS-JEDI is done
Plans continue moving forward to adopt model variable naming convention in JEDI
While planning the code sprint for this project, lots of ways to improve the code and make implementation much easier were found
Update to background error covariance training suite to be more robust and flexible
Snow depth DA is now getting reasonable results and runs in less than 3 minutes
In final testing for GFSv17
Option for different background covariance from SABER blocks
Goal for this year: multiscale background error covariance
using the spectral analytical filter block for ensemble covariance localization is in progress
Another big goal for this year is EDA with SkyLab-GEOS, aiming to emulate the current operational system for the high resolution system
Simple variational case is complete
Next steps is making it more sophisticated with more obs and better covariances, and do EDA later this year
The HTLM is showing good reduction in linearization errors
Another theme for this year: continuous DA
Introduce new options in variational DA
Work is in progress on capability to add obs added since last outer loop in the next outer loop
Plans for Q2:
Generic development in:
IODA: ObsDataFrame integration for more flexible and efficient data storage
VADER: improvements to linear variable change for more flexibility and use in SABER
Multiscale B: more extensive testing of spectral analytical filter with mpas-jedi (and fv3-jedi?); initial tests with multiscale covariances with mpas-jedi.
Skylab-GEOS: 3D-Var with most observations, hybrid B (GSI + BUMP)
Continuous DA: ability to add newly arrived observations
Congratulations to the entire JCSDA staff and all of our partners and in-kinds for these big steps forward! Keep an eye out for upcoming developments with space weather prediction, visual reflectance DA, and coupled DA!