Object-Based Evaluation of a Numerical Weather Prediction Model’s Performance through Forecast Storm Characteristic Analysis

In this study, we evaluate the ability of the High-Resolution Rapid Refresh (HRRR) model to forecast cloud characteristics through comparison of observed and simulated satellite brightness temperatures (BTs) and radar reflectivity during different weather phenomena in December 2021: the Mayfield, Kentucky, tornado on 11 December, a heavy snow event in Minnesota from 10 to 11 December, and the Midwest derecho on 15 December. This is done to illustrate the importance of examining model accuracy across a range of weather phenomena. Observation and forecast objects were created using the Method for Object-Based Diagnostic Evaluation (MODE). HRRR accurately depicted the spatial displacements between observation cloud (defined using BTs) and radar reflectivity objects, namely, the centers of cloud objects are to the east of the radar objects for the tornado and derecho events, and generally west of the radar objects for the snow event. However, HRRR had higher (less intense) simulated BTs and higher (more intense) radar reflectivity than the observations for the tornado event. Simulated radar reflectivity is higher and BTs are lower than the observations during the middle of the snow event. Also, simulated radar reflectivity is higher and BTs are lower than the observations during the derecho event. Of the three weather events, the HRRR forecasts are most accurate for the snow event, based on the object-based threat score, followed by the derecho and tornado events. The tornado event has lower accuracy because matches between paired simulated and observation objects are worse than for the snow event, with less similarity in size forecast objects and greater distance between paired object centers. Significance Statement The purpose of this study is to assess the accuracy of forecast cloud and radar objects, defined using simulated satellite brightness temperatures and radar reflectivity, from the High-Resolution Rapid Refresh (HRRR) model. This assessment was conducted for a tornado, snow, and derecho event from December 2021. Results from these three events indicate that the HRRR model accurately represents the observed displacement between the center of cloud and radar objects for the tornado and derecho events, and is the most accurate overall for the snow event.

Strong wind events can cause severe economic loss and societal impacts. Peak winds over Wyoming and Colorado occur during the wintertime months, and in the first part of this two-part series, it was shown that the High-Resolution Rapid Refresh (HRRR) model displays large negative biases with respect to strong wind events. In this part of the study, we address two questions: 1) does increasing the horizontal resolution improve the representation of strong wind events over this region, and 2) are the biases in HRRR-forecasted winds related to the selected planetary boundary layer (PBL), surface layer (SL), and/or land surface model (LSM) parameterizations? We conduct Weather Research and Forecasting (WRF) Model simulations to address these two main questions. Increasing the horizontal resolution leads to a small improvement in the simulation of strong wind speeds over the complex terrain of Wyoming and Colorado. In general, changes in the PBL, SL, and LSM parameterizations show much larger changes in simulated wind speeds compared to increasing the model resolution alone. Specifically, changing from the Mellor–Yamada–Nakanishi–Niino scheme to the Mellor–Yamada–Janjić PBL and SL schemes results in nearly no change in the r2 values, but there is a decrease in the magnitude of the strong wind speed bias (from −12.52 to −10.16 m s−1). We attribute these differences to differences in the diagnosis of surface wind speeds and mixing in the boundary layer. Further analysis is conducted to determine the value of 1-km forecasts of strong winds compared with wind speed diagnostics commonly used by the National Weather Service. Significance Statement Motivated by prior studies showing low skill in the prediction of strong winds using state-of-the-art weather forecast models, in this study, we aim to investigate two questions: 1) does increasing the horizontal resolution improve the prediction of strong wind events over the complex terrain of Wyoming and Colorado, and 2) are the biases in High-Resolution Rapid Refresh (HRRR) forecasted winds related to the selected planetary boundary layer, surface layer, and/or land surface model parameterizations? We find that increasing the horizontal resolution provides a slight improvement in the prediction of strong winds. Further, considerable improvement in the prediction of strong winds is found for varying boundary layer, surface layer, and land surface parameterizations.

A technique for model initialization using three-dimensional radar reflectivity data has been developed and applied within the NOAA 13-km Rapid Refresh (RAP) and 3-km High-Resolution Rapid Refresh (HRRR) regional forecast systems. This technique enabled the first assimilation of radar reflectivity data for operational NOAA forecast models, critical especially for more accurate short-range prediction of convective storms. For the RAP, the technique uses a diabatic digital filter initialization (DFI) procedure originally deployed to control initial inertial gravity wave noise. Within the forward-model integration portion of diabatic DFI, temperature tendencies obtained from the model cloud/precipitation processes are replaced by specified latent heating–based temperature tendencies derived from the three-dimensional radar reflectivity data, where available. To further refine initial conditions for the convection-allowing HRRR model, a similar procedure is used in the HRRR, but without DFI. Both of these procedures, together called the “Radar-LHI” (latent heating initialization) technique, have been essential for initialization of ongoing precipitation systems, especially convective systems, within all NOAA operational versions of the 13-km RAP and 3-km HRRR models extending through the latest implementation upgrade at NCEP in 2020. Application of the latent heat–derived temperature tendency induces a vertical circulation with low-level convergence and upper-level divergence in precipitation systems. Retrospective tests of the Radar-LHI technique show significant improvement in short-range (0–6 h) precipitation system forecasts, as revealed by reflectivity verification scores. Results presented document the impact on HRRR reflectivity forecasts of the radar reflectivity initialization technique applied to the RAP alone, HRRR alone, and both the RAP and HRRR. Significance Statement The large forecast uncertainty of convective situations, even at short lead times, coupled with the hazardous weather they produce, makes convective storm prediction one of the most significant short-range forecast challenges confronting the operational numerical weather prediction community. Prediction of heavy precipitation events also requires accurate initialization of precipitation systems. An innovative assimilation technique using radar reflectivity data to initialize NOAA operational weather prediction models is described. This technique, which uses latent heating specified from radar reflectivity (and can accommodate lightning data and other convection/precipitation indicators), was first implemented in 2009 at NOAA/NCEP and continues to be used in 2022 in the NCEP-operational RAP and HRRR models, making it a backbone of the NOAA rapidly updated numerical weather prediction capability.

Forecasts from numerical weather prediction (NWP) models play a critical role in many sectors of the U.S. economy. Improvements to operational NWP model forecasts are generally assumed to provide significant economic savings through better decision-making. But is this true? Since 2014, several new versions of the High-Resolution Rapid Refresh (HRRR) model were released into operation within the National Weather Service. Practically, forecasts have an economic impact only if they lead to a different action than what would be taken under an alternative information set. And in many sectors, these decisions only need to be considered during certain weather conditions. We estimate the economic impacts of improvements made to the HRRR, using 12-h wind, precipitation, and temperature forecasts in several cases where they can have “economically meaningful” behavioral consequences. We examine three different components of the U.S. economy where such information matters: 1) better integration of wind energy resources into the electric grid, 2) increased worker output due to better precipitation forecasts that allow workers to arrive to their jobs on time, and 3) better decisions by agricultural producers in preparing for freezing conditions. These applications demonstrate some of the challenges in ascertaining the economic impacts of improved weather forecasts, including highlighting key assumptions that must be made to make the problem tractable. For these sectors, we demonstrate that there was a marked economic gain for the United States between HRRR versions 1 and 2 and a smaller, but still appreciable economic gain between versions 2 and 3.

The hourly updating High-Resolution Rapid Refresh (HRRR) model is evaluated with regard to its ability to predict the areal extent of cold-season precipitation and accurately depict the timing and location of regions of snow, rain, and mixed-phase precipitation on the ground. Validation of the HRRR forecasts is performed using observations collected by the Automated Surface Observing System (ASOS) stations across the eastern two-thirds of the United States during the 2010–11 cold season. The results show that the HRRR is able to reliably forecast precipitation extent during the cold season. In particular, the location and areal extent of both snow and rain are very well predicted. Depiction of rain-to-snow transitions and freezing rain is reasonably good; however, the associated evaluation scores are significantly lower than for either snow or rain. The analyses suggest the skill in accurately depicting precipitation extent and phase (i.e., rain, snow, and mixed phase) depends on the size and organization of a weather system. Typically, larger synoptically forced weather systems are better predicted than smaller weather systems, including the associated rain-to-snow transition or freezing-rain areas. Offsets in space or time (i.e., causing misses and false alarms) have a larger effect on the model performance for smaller weather systems.

A comprehensive assessment of the forecast skill of various meteorological indices over the Indian region from different numerical weather prediction (NWP) models is lacking in the literature. In this study the performance of four NWP models, namely Global Ensemble Forecast System (GEFSv12), European Center for Medium Range Forecasting (ECMWF), Climate Forecast System (CFSv2) and Indian Institute of Tropical Meteorology (IITM) towards forecasting of precipitation, temperature and associated meteorological indices, is evaluated at short to medium timescales across the Indian region. Further, the effect of ocean atmospheric (OA) oscillations on the precipitation/temperature forecast skill from the different NWP models is also assessed. Results show that the NWP models are better in predicting the meteorological indices than the quantitative forecasts. The ECMWF model was found to be the best for PCP forecasting with CFSv2 performing poorly. For temperature the GEFSv12 model performance was the lowest, compared to rest of the models. The models show poor skill in forecasting monsoon season precipitation compared to non‐monsoon and the temperature forecasts from the NWP models are particularly poor for the northern basins. Skilful temperature forecasts are observed in the Northwestern, Indo‐Gangetic and Central basins for the CFSv2 and ECMWF models. The forecast skill of precipitation indices are higher in the northwestern, central and Indo‐Gangetic basins compared to the rest. The skill of precipitation indices, namely rainy days, extreme rainy days and consecutive wet days, is higher during the monsoon seasons while the prediction skill of consecutive dry days is higher during the non‐monsoon season. OA analysis revealed that the ENSO phases have dominant effect on the forecast skill of the precipitation only. The temperature and meteorological indices forecasts are not affected significantly by the OA phases. The outcomes of this study have implications towards irrigation scheduling and water resources management decision making in India.

Abstract. The current operational very short-term and short-term quantitative precipitation forecast (QPF) at the Meteorological Service of Catalonia (SMC) is made by three different methodologies: Advection of the radar reflectivity field (ADV), Identification, tracking and forecasting of convective structures (CST) and numerical weather prediction (NWP) models using observational data assimilation (radar, satellite, etc.). These precipitation forecasts have different characteristics, lead time and spatial resolutions. The objective of this study is to combine these methods in order to obtain a single and optimized QPF at each lead time. This combination (blending) of the radar forecast (ADV and CST) and precipitation forecast from NWP model is carried out by means of different methodologies according to the prediction horizon. Firstly, in order to take advantage of the rainfall location and intensity from radar observations, a phase correction technique is applied to the NWP output to derive an additional corrected forecast (MCO). To select the best precipitation estimation in the first and second hour (t+1 h and t+2 h), the information from radar advection (ADV) and the corrected outputs from the model (MCO) are mixed by using different weights, which vary dynamically, according to indexes that quantify the quality of these predictions. This procedure has the ability to integrate the skill of rainfall location and patterns that are given by the advection of radar reflectivity field with the capacity of generating new precipitation areas from the NWP models. From the third hour (t+3 h), as radar-based forecasting has generally low skills, only the quantitative precipitation forecast from model is used. This blending of different sources of prediction is verified for different types of episodes (convective, moderately convective and stratiform) to obtain a robust methodology for implementing it in an operational and dynamic way.

Researchers test the High-Resolution Rapid Refresh model with real-life observational data to evaluate forecast accuracy.

BACKGROUND: Wildland fire smoke contains large amounts of PM2.5 that can traverse tens to hundreds of kilometers, resulting in significant deterioration of air quality and excess mortality and morbidity in downwind regions. Estimating PM2.5 levels while considering the impact of wildfire smoke has been challenging due to the lack of ground monitoring coverage near the smoke plumes. AIM: We aim to estimate total PM2.5 concentration during the Camp Fire episode, the deadliest wildland fire in California history. METHODS: Our random forest (RF) model combines calibrated low-cost sensor data (PurpleAir) with regulatory monitor measurements (Air Quality System, AQS) to bolster ground observations, Geostationary Operational Environmental Satellite-16 (GOES-16)’s high temporal resolution to achieve hourly predictions, and oversampling techniques (Synthetic Minority Oversampling Technique, SMOTE) to reduce model underestimation at high PM2.5 levels. In addition, meteorological fields at 3 km resolution from the High-Resolution Rapid Refresh model and land use variables were also included in the model. RESULTS: Our AQS-only model achieved an out of bag (OOB) R2 (RMSE) of 0.84 (12.00 μg/m3) and spatial and temporal cross-validation (CV) R2 (RMSE) of 0.74 (16.28 μg/m3) and 0.73 (16.58 μg/m3), respectively. Our AQS + Weighted PurpleAir Model achieved OOB R2 (RMSE) of 0.86 (9.52 μg/m3) and spatial and temporal CV R2 (RMSE) of 0.75 (14.93 μg/m3) and 0.79 (11.89 μg/m3), respectively. Our AQS + Weighted PurpleAir + SMOTE Model achieved OOB R2 (RMSE) of 0.92 (10.44 μg/m3) and spatial and temporal CV R2 (RMSE) of 0.84 (12.36 μg/m3) and 0.85 (14.88 μg/m3), respectively. CONCLUSIONS: Hourly predictions from our model may aid in epidemiological investigations of intense and acute exposure to PM2.5 during the Camp Fire episode. KEYWORDS: PM2.5, GOES16, Wildland fire, Remote sensing, AOD, SMOTE, Weighted Random Forest

Monitoring the fuel moisture content (FMC) of 10 h dead vegetation is crucial for managing and mitigating the impact of wildland fires. The combination of in situ FMC observations, numerical weather prediction (NWP) models, and satellite retrievals has facilitated the development of machine learning (ML) models to estimate 10 h dead FMC retrievals over the contiguous US (CONUS). In this study, ML models were trained using variables from the National Water Model, the High-Resolution Rapid Refresh (HRRR) NWP model, and static surface properties, along with surface reflectances and land surface temperature (LST) retrievals from the Visible Infrared Imaging Radiometer Suite (VIIRS) instrument on the Suomi-NPP satellite system. Extensive hyper-parameter optimization resulted in skillful FMC models compared to a daily climatography RMSE (+44%) and an hourly climatography RMSE (+24%). Notably, VIIRS retrievals played a significant role as predictors for estimating 10 h dead FMC, demonstrating their importance as a group due to their high band correlation. Conversely, individual predictors within the HRRR group exhibited relatively high importance according to explainability techniques. Removing both HRRR and VIIRS retrievals as model inputs led to a significant decline in performance, particularly with worse RMSE values when excluding VIIRS retrievals. The importance of the VIIRS predictor group reinforces the dynamic relationship between 10 h dead fuel, the atmosphere, and soil moisture. These findings underscore the significance of selecting appropriate data sources when utilizing ML models for FMC prediction. VIIRS retrievals, in combination with selected HRRR variables, emerge as critical components in achieving skillful FMC estimates.

<p indent="0mm">In real-time operations, the minutely/hourly updated high-resolution rapid refresh (HRRR) system is one of the most expensive numerical weather prediction (NWP) models. Based on a twenty-member HRRR-time-lagged-ensemble (HRRR-TLE) system developed from two real-time convection-permitting HRRR models, CMA-GD(R3) and CMA-SH3, from the China Meteorological Administration (CMA), this study proposes an optimized probability-matching (OPM) technique to improve 0−12 h quantitative precipitation forecasts (QPFs) based on the correlation and error relationships between ensemble forecasts and observations during the training window. Then, a series of sensitivity experiments using different cost functions and optimized ratios was conducted to further improve OPM predictions. The results indicate that: (1) In the HRRR-TLE system, there is no always optimal member in both weak rain and severe rain forecasts, as measured by the equitable threat score (ETS) and bias extent (BE) at four thresholds (1+, 5+, 10+, and 20+ mm h^–1; e.g., “1+” means ≥ 1). (2) Compared with the HRRR-TLE system, the QPFs generated by the traditional PM technique showed a notable increase in ETS and a decrease in BE at all of the above thresholds. Compared with the traditional probability-matching method (PM), OPM can generate more skillful forecasts on both spatial representations and rain rates by using the sliding-weight method and optimized ensembles, respectively. (3) In particular, in the 20+ mm h^–1 forecasts, which are often difficult to predict, the ETS of the optimal OPM test, with a 20% optimization ratio and symmetric mean absolute percentage error cost function, increased by 64.6%, and the BE decreased by 5.7%, relative to PM. Moreover, OPM shows good stability in both daytime and nighttime periods.

The High-Resolution Rapid Refresh (HRRR) is a convection-allowing implementation of the Advanced Research version of the Weather Research and Forecasting (WRF-ARW) Model with hourly data assimilation that covers the conterminous United States and Alaska and runs in real time at the NOAA/National Centers for Environmental Prediction (NCEP). Implemented operationally at NOAA/NCEP in 2014, the HRRR features 3-km horizontal grid spacing and frequent forecasts (hourly for CONUS and 3-hourly for Alaska). HRRR initialization is designed for optimal short-range forecast skill with a particular focus on the evolution of precipitating systems. Key components of the initialization are radar-reflectivity data assimilation, hybrid ensemble-variational assimilation of conventional weather observations, and a cloud analysis to initialize stratiform cloud layers. From this initial state, HRRR forecasts are produced out to 18 h every hour, and out to 48 h every 6 h, with boundary conditions provided by the Rapid Refresh system. Between 2014 and 2020, HRRR development was focused on reducing model bias errors and improving forecast realism and accuracy. Improved representation of the planetary boundary layer, subgrid-scale clouds, and land surface contributed extensively to overall HRRR improvements. The final version of the HRRR (HRRRv4), implemented in late 2020, also features hybrid data assimilation using flow-dependent covariances from a 3-km, 36-member ensemble (“HRRRDAS”) with explicit convective storms. HRRRv4 also includes prediction of wildfire smoke plumes. The HRRR provides a baseline capability for evaluating NOAA’s next-generation Rapid Refresh Forecast System, now under development. Significance Statement NOAA’s operational hourly updating, convection-allowing model, the High-Resolution Rapid Refresh (HRRR), is a key tool for short-range weather forecasting and situational awareness. Improvements in assimilation of weather observations, as well as in physics parameterizations, have led to improvements in simulated radar reflectivity and quantitative precipitation forecasts since the initial implementation of HRRR in September 2014. Other targeted development has focused on improved representation of the diurnal cycle of the planetary boundary layer, resulting in improved near-surface temperature and humidity forecasts. Additional physics and data assimilation changes have led to improved treatment of the development and erosion of low-level clouds, including subgrid-scale clouds. The final version of HRRR features storm-scale ensemble data assimilation and explicit prediction of wildfire smoke plumes.

Relatively little is known about how topography affects convective storms. The first step toward understanding these effects is to investigate how topography affects storm environments. Unfortunately, the effects of topography on convective environments are not easily observed directly. Instead, it is necessary to resort to using output from the High-Resolution Rapid Refresh (HRRR). The HRRR’s 3-km grid spacing can resolve some of the larger-scale topographic effects. Popular convective storm forecasting parameters obtained from the HRRR are averaged on convective days from February to September 2013–15. It is surmised that most of the day-to-day variability attributable to synoptic- and mesoscale meteorological influences is removed by averaging; the remaining horizontal heterogeneity in parameters related to instability and vertical wind shear is due to the hemispheric-scale meridional temperature and pressure gradient, and likely also topographic influences, especially where recurring longitudinal variations in instability, wind shear, etc. are found. Anomalies are sensitive to the ambient low-level wind direction (i.e., whether winds are locally blowing upslope or downslope), especially for parameters that depend on the low-level vertical shear. The statistical significance of local maxima and minima is demonstrated by comparing the amplitudes of the anomalies to bootstrapped estimates of the standard errors.

Tropical cyclone tornadoes (TCTORs) are a hazard to life and property during landfalling tropical cyclones (TCs). The threat is often spread over a wide area within the TC envelope and must be continually evaluated as the TC moves inland and dissipates. To anticipate the risk of TCTORs, forecasters may use high-resolution, rapidly updating model analyses and short-range forecasts such as the Rapid Refresh (RAP) and High-Resolution Rapid Refresh (HRRR), and an ingredients-based approach similar to that used for forecasting continental midlatitude tornadoes. Though RAP and HRRR errors have been identified in typical midlatitude convective environments, this study evaluates the performance of the RAP and the HRRR within the TC envelope, with particular attention given to sounding-derived parameters previously identified as useful for TCTOR forecasting. A sample of 1730 observed upper-air soundings is sourced from 13 TCs that made landfall along the U.S. coastline between 2017 and 2019. The observed soundings are paired with their corresponding model gridpoint soundings from the RAP analysis, RAP 12-h forecast, and HRRR 12-h forecast. Model errors are calculated for both the raw sounding variables of temperature, dewpoint, and wind speed, as well as for the selected sounding-derived parameters. Results show a moist bias that worsens with height across all model runs. There are also statistically significant underpredictions in stability-related parameters such as convective available potential energy (CAPE) and kinematic parameters such as vertical wind shear.

In 2015 the U.S. Department of Energy (DOE) initiated a 4-yr study, the Second Wind Forecast Improvement Project (WFIP2), to improve the representation of boundary layer physics and related processes in mesoscale models for better treatment of scales applicable to wind and wind power forecasts. This goal challenges numerical weather prediction (NWP) models in complex terrain in large part because of inherent assumptions underlying their boundary layer parameterizations. The WFIP2 effort involved the wind industry, universities, the National Oceanographic and Atmospheric Administration (NOAA), and the DOE’s national laboratories in an integrated observational and modeling study. Observations spanned 18 months to assure a full annual cycle of continuously recorded observations from remote sensing and in situ measurement systems. The study area comprised the Columbia basin of eastern Washington and Oregon, containing more than 6 GW of installed wind capacity. Nests of observational systems captured important atmospheric scales from mesoscale to NWP subgrid scale. Model improvements targeted NOAA’s High-Resolution Rapid Refresh (HRRR) model to facilitate transfer of improvements to National Weather Service (NWS) operational forecast models, and these modifications have already yielded quantitative improvements for the short-term operational forecasts. This paper describes the general WFIP2 scope and objectives, the particular scientific challenges of improving wind forecasts in complex terrain, early successes of the project, and an integrated approach to archiving observations and model output. It provides an introduction for a set of more detailedBAMSpapers addressing WFIP2 observational science, modeling challenges and solutions, incorporation of forecasting uncertainty into decision support tools for the wind industry, and advances in coupling improved mesoscale models to microscale models that can represent interactions between wind plants and the atmosphere.