Application of Forecast Verification Science to Operational

January 13, 2018 | Author: Anonymous | Category: Math, Statistics And Probability, Statistics
Share Embed Donate


Short Description

Download Application of Forecast Verification Science to Operational...

Description

Application of Forecast Verification Science to Operational River Forecasting in the National Weather Service Julie Demargne, James Brown, Yuqiong Liu and D-J Seo UCAR

NROW, November 4-5, 2009

Approach to river forecasting Observations

Forecasters

Models Input forecasts

EVAPOTRANSPIRATION

INFILTRATION FREE

PERCOLATION

LOWER ZONE

Users

TENSION

UPPER ZONE

PRIM ARY FREE

RESERVED

INTERFLOW

SURFACE RUNOFF

TENSION TENSION SUPPLEM ENTAL FREE

RESERVED

BASEFLOW

SUBSURFACE OUTFLOW

DIRECT RUNOFF

Forecast products

Forecasters 2

Where is the …? In the past Verification ?? ?

• Limited verification of hydrologic forecasts

• How good are the forecasts for application X?

3

Where is the …? Now Verification !!!

Papers

Verification Experts

Verification Products Verification Systems 4

Hydrologic forecasting: a multi-scale problem

National

Forecast group

Major river system

River basin with river forecast points

Headwater basin with radar rainfall grid

High resolution flash flood basins

Hydrologic forecasts must be verified consistently across all spatial scales and resolutions.

5

Hydrologic forecasting: a multi-scale problem Forecast Uncertainty

Years

Seasons Months

Forecast Lead Time

Weeks Days Hours Minutes Protection of Life & Property

Benefits Hydropower

Flood Mitigation & Navigation

Recreation

Agriculture

Ecosystem

Reservoir Control

State/Local Planning

Health

Environment

Commerce

Seamless probabilistic water forecasts are required for all lead times and all users; so is verification information. 6

Need for hydrologic forecast verification • In 2006, NRC recommended NWS expand verification of its uncertainty products and make it easily available to all users in near real time

Users decide whether to take action with risk-based decision Must educate users on how to interpret forecast and verification info

7

River forecast verification service

http://www.nws.noaa.gov/oh/ rfcdev/docs/ Final_Verification_Report.pdf

http://www.nws.noaa.gov/oh/rfcdev/docs/ NWS-Hydrologic-Forecast-VerificationTeam_Final-report_Sep09.pdf.pdf

8

River forecast verification service • To help us answer  How good are the forecasts for application X?  What are the strengths and weaknesses of the forecasts?  What are the sources of error and uncertainty in the

forecasts?  How are new science and technology improving the

forecasts and the verifying observations?  What should be done to improve the forecasts?  Do forecasts help users in their decision making?

9

River forecast verification service River forecasting system Observations

Verification systems

Models Input forecasts

Forecasters Users

EVAPOTRANSPIRATION

TENSION

UPPER ZONE

INFILTRATION FREE

PERCOLATION

LOWER ZONE

PRIM ARY FREE

INTERFLOW

SURFACE RUNOFF

TENSION TENSION SUPPLEM ENTAL FREE

RESERVED

RESERVED

BASEFLOW

SUBSURFACE OUTFLOW

Users

DIRECT RUNOFF

Forecast products

Verification products

10

River forecast verification service •

Verification Service within Community Hydrologic Prediction System (CHPS) to:  Compute metrics  Display data & metrics  Disseminate data & metrics

 Provide real-time access to metrics  Analyze uncertainty and error in forecasts  Track performance

11

Verification challenges • Verification is useful if the information generated leads to decisions about the forecast/system being verified  Verification needs to be user oriented

• No single verification measure provides complete information about the quality of a forecast product  Several verification metrics and products are needed

• To facilitate communication of forecast quality, common verification practices and products are needed from weather and climate forecasts to water forecasts  Collaborations between meteorology and hydrology communities

are needed (e.g., Thorpex-Hydro, HEPEX) 12

Verification challenges: two classes of verification • Diagnostic verification:  to diagnose and improve model performance  done off-line with archived forecasts or hindcasts to analyze

forecast quality relative to different conditions/processes

• Real-time verification:  to help forecasters and users make decisions in real-time  done in real-time (before the verifying observation occurs)

using information from historical analogs and/or past forecasts and verifying observations under similar conditions 13

Diagnostic verification products • Key verification metrics for 4 levels of information for single-valued and probabilistic forecasts 1. Observations-forecasts comparisons (scatter plots, box plots, time series plots) 2. Summary verification (e.g. MAE/Mean CRPS, skill score)

3. More detailed verification (e.g. measures of reliability, resolution, discrimination, correlation, results for specific conditions) 4. Sophisticated verification (e.g. for specific events with ROC)

To be evaluated by forecasters and forecast users 14

Diagnostic verification products

Forecast value

• Examples for level 1: scatter plot, box-and-whiskers plot

User-defined threshold

Observed value 15

Diagnostic verification products • Examples for level 1: box-and-whiskers plot ‘Errors’ for one forecast Max.

90% 80%

Median

20% 10%

Forecast error (forecast - observed) [mm]

American River in California – 24-hr precipitation ensembles (lead day 1) Zero error line

“Blown” forecasts

High bias

Low bias

Min.

Observed daily total precipitation [mm]

16

Diagnostic verification products • Examples for level 2: skill score maps by months January

April

October

Smaller score, better 17

Diagnostic verification products • Examples for level 3: more detailed plots Score

Performance under different conditions

Score

Performance for different months 18

Diagnostic verification products • Examples for level 4: event specific plots Event: > 85th percentile from observed distribution

Reliability

Perfect

Predicted Probability

Discrimination

Probability of Detection POD

Observed frequency

Perfect

Probability of False Detection POFD 19

Diagnostic verification products • Examples for level 4: user-friendly spread-bias plot “Hit rate” = 90%

60% of time, observation should fall in window covering middle 60% (i.e. median ±30%)

60%

“Underspread”

20

Diagnostic verification analyses • Analyze any new forecast process with verification • Use different temporal aggregations  Analyze verification statistic as a function of lead time  If similar performance across lead times, data can be pooled

• Perform spatial aggregation carefully  Analyze results for each basin and results plotted on spatial maps  Use normalized metrics (e.g. skill scores)

 Aggregate verification results across basins with similar hydrologic

processes (e.g. by response time)

• Report verification scores with sample size  In the future, confidence intervals 21

Diagnostic verification analyses • Evaluate forecast performance under different conditions w/ time conditioning: by month, by season w/ atmospheric/hydrologic conditioning: – low/high probability threshold – absolute thresholds (e.g., PoP, Flood Stage)

Check that sample size is not too small

• Analyze sources of uncertainty and error Verify forcing input forecasts and output forecasts

For extreme events, verify both stage and flow Sensitivity analysis to be set up at all RFCs: 1) what is the optimized QPF horizon for hydrologic forecasts? 2) do run-time modifications made on the fly improve forecasts?

22

Diagnostic verification software • Interactive Verification Program (IVP) developed at OHD: verifies single-valued forecasts at given locations/areas

23

Diagnostic verification software • Ensemble Verification System (EVS) developed at OHD: verifies ensemble forecasts at given locations/areas

24

Dissemination of diagnostic verification • Example: WR water supply website http://www.nwrfc.noaa.gov/westernwater/ Data Visualization

Error •MAE, RMSE •Conditional on lead time, year

Skill •Skill relative to Climatology •Conditional

Categorical •FAR, POD, contingency table (based on climatology or user definable) 25

Dissemination of diagnostic verification • Example: OHRFC bubble plot online http://www.erh.noaa.gov/ohrfc/bubbles.php

26

Real-time verification • How good could the ‘live’ forecast be? Live forecast Observations

27

Real-time verification • Select analogs from a pre-defined set of historical events and compare with ‘live’ forecast

Analog 3

Analog 2

Analog 1

Observed Live forecast Analog Observed Analog Forecast

“Live forecast for Flood is likely to be too high” 28

Real-time verification • Adjust ‘live’ forecast based on info from the historical analogs Live forecast

What happened

Live forecast was too high 29

Real-time verification • Example for ensemble forecasts Live forecast (L) Analog forecasts (H): μH = μL ± 1.0˚C

Temperature (oF)

Analog observations

“Day 1 forecast is probably too high” Forecast lead day

30

Real-time verification • Build analog query prototype using multiple criteria  Seeking analogs for precipitation: “Give me past forecasts for

the 10 largest events relative to hurricanes for this basin.”  Seeking analogs for temperature: “Give me all past forecasts

with lead time 12 hours whose ensemble mean was within 5% of the live ensemble mean.”  Seeking analogs for flow: “Give me all past forecasts with lead

times of 12-48 hours whose probability of flooding was >=0.95, where the basin-averaged soil-moisture was > x and the immediately prior observed flow exceeded y at the forecast issue time”.

Requires forecasters’ input!

31

Outstanding science issues • • • • • •

Define meaningful reference forecasts for skill scores



Account for observational error (measurement and representativeness errors) and rating curve error



Account for non-stationarity (e.g., climate change)

Separate timing error and amplitude error in forecasts

Verify rare events and specify sampling uncertainty in metrics Analyze sources of uncertainty and error in forecasts Consistently verify forecasts on multiple space and time scales

Verify multivariate forecasts (issued at multiple locations and for multiple time steps) by accounting for statistical dependencies

32

Verification service development OHD OCWWS NCEP

Forecasters Users

Academia Forecast agencies COMET-OHD-OCWWS OHD-NCEP Thorpex-Hydro project collaboration on training Private OHD-Deltares collaboration for CHPS enhancements

HEPEX Verification Test Bed (CMC, Hydro-Quebec, ECMWF)

33

Looking ahead •



2012: 

Info on quality of forecast service available online



real-time and diagnostic verification implemented in CHPS



RFC verification standard products available online along with forecasts

2015: 

Leveraging grid-based verification tools

34

Thank you Questions?

FORECASTER

FORECASTER

[email protected] 35

Extra slide

36

Diagnostic verification products •

Key verification metrics from NWS Verification Team report

37

View more...

Comments

Copyright � 2017 NANOPDF Inc.
SUPPORT NANOPDF