Assessing Hourly Precipitation Forecast Skill with the Fractions Skill Score
|
摘要
Statistical methods for category(yes/no) forecasts, such as the Threat Score, are typically used in the verification of precipitation forecasts. However, these standard methods are affected by the so-called "double-penalty" problem caused by slight displacements in either space or time with respect to the observations. Spatial techniques have recently been developed to help solve this problem. The fractions skill score(FSS), a neighborhood spatial verification method, directly compares the fractional coverage of events in windows surrounding the observations and forecasts.We applied the FSS to hourly precipitation verification by taking hourly forecast products from the GRAPES(Global/Regional Assimilation Prediction System) regional model and quantitative precipitation estimation products from the National Meteorological Information Center of China during July and August 2016, and investigated the difference between these results and those obtained with the traditional category score. We found that the model spin-up period affected the assessment of stability. Systematic errors had an insignificant role in the fraction Brier score and could be ignored. The dispersion of observations followed a diurnal cycle and the standard deviation of the forecast had a similar pattern to the reference maximum of the fraction Brier score. The coefficient of the forecasts and the observations is similar to the FSS; that is, the FSS may be a useful index that can be used to indicate correlation.Compared with the traditional skill score, the FSS has obvious advantages in distinguishing differences in precipitation time series, especially in the assessment of heavy rainfall.
Statistical methods for category(yes/no) forecasts, such as the Threat Score, are typically used in the verification of precipitation forecasts. However, these standard methods are affected by the so-called "double-penalty" problem caused by slight displacements in either space or time with respect to the observations. Spatial techniques have recently been developed to help solve this problem. The fractions skill score(FSS), a neighborhood spatial verification method, directly compares the fractional coverage of events in windows surrounding the observations and forecasts.We applied the FSS to hourly precipitation verification by taking hourly forecast products from the GRAPES(Global/Regional Assimilation Prediction System) regional model and quantitative precipitation estimation products from the National Meteorological Information Center of China during July and August 2016, and investigated the difference between these results and those obtained with the traditional category score. We found that the model spin-up period affected the assessment of stability. Systematic errors had an insignificant role in the fraction Brier score and could be ignored. The dispersion of observations followed a diurnal cycle and the standard deviation of the forecast had a similar pattern to the reference maximum of the fraction Brier score. The coefficient of the forecasts and the observations is similar to the FSS; that is, the FSS may be a useful index that can be used to indicate correlation.Compared with the traditional skill score, the FSS has obvious advantages in distinguishing differences in precipitation time series, especially in the assessment of heavy rainfall.
Statistical methods for category(yes/no) forecasts, such as the Threat Score, are typically used in the verification of precipitation forecasts. However, these standard methods are affected by the so-called "double-penalty" problem caused by slight displacements in either space or time with respect to the observations. Spatial techniques have recently been developed to help solve this problem. The fractions skill score(FSS), a neighborhood spatial verification method, directly compares the fractional coverage of events in windows surrounding the observations and forecasts.We applied the FSS to hourly precipitation verification by taking hourly forecast products from the GRAPES(Global/Regional Assimilation Prediction System) regional model and quantitative precipitation estimation products from the National Meteorological Information Center of China during July and August 2016, and investigated the difference between these results and those obtained with the traditional category score. We found that the model spin-up period affected the assessment of stability. Systematic errors had an insignificant role in the fraction Brier score and could be ignored. The dispersion of observations followed a diurnal cycle and the standard deviation of the forecast had a similar pattern to the reference maximum of the fraction Brier score. The coefficient of the forecasts and the observations is similar to the FSS; that is, the FSS may be a useful index that can be used to indicate correlation.Compared with the traditional skill score, the FSS has obvious advantages in distinguishing differences in precipitation time series, especially in the assessment of heavy rainfall.
引文
Ahijevych,D.,E.Gilleland,B.G.Brown,et al.,2009:Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts.Wea.Forecasting,24,1485-1497,doi:10.1175/2009WAF2222298.1.
Atger,F.,2001:Verification of intense precipitation forecasts from single models and ensemble prediction systems.Nonlin.Proc.Geophys.,8,401-417,doi:10.5194/npg-8-401-2001.
Baldwin,M.E.,and J.S.Kain,2006:Sensitivity of several performance measures to displacement error,bias,and event frequency.Wea.Forecasting,21,636-648,doi:10.1175/WAF933.1.
Brill,K.F.,and F.Mesinger,2009:Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores.Wea.Forecasting,24,1748-1754,doi:10.1175/2009WAF2222272.1.
Casati,B.,2010:New developments of the intensity-scale technique within the Spatial Verification Methods Intercomparison Project.Wea.Forecasting,25,113-143,doi:10.1175/2009WAF2222257.1.
Casati,B.,G.Ross,and D.B.Stephenson,2004:A new intensityscale approach for the verification of spatial precipitation forecasts.Meteor.Appl.,11,141-154,doi:10.1017/S1350482704001239.
Casati,B.,L.J.Wilson,D.B.Stephenson,et al.,2008:Forecast verification:Current status and future directions.Meteor.Appl.,15,3-18,doi:10.1002/(ISSN)1469-8080.
Chen,J.,Y.Wang,L.Li,et al.,2013:A unified verification system for operational models from Regional Meteorological Centres of China Meteorological Administration.Meteor.Appl.,20,140-149,doi:10.1002/met.1406.
Davis,C.A.,B.G.Brown,R.Bullock,et al.,2009:The method for Object-Based Diagnostic Evaluation(MODE)applied to numerical forecasts from the 2005 NSSL/SPC Spring Program.Wea.Forecasting,24,1252-1267,doi:10.1175/2009WAF2222241.1.
Ebert,E.E.,2008:Fuzzy verification of high-resolution gridded forecasts:A review and proposed framework.Meteor.Appl.,15,51-64,doi:10.1002/(ISSN)1469-8080.
Ebert,E.E.,2009:Neighborhood verification:A strategy for rewarding close forecasts.Wea.Forecasting,24,1498-1510,doi:10.1175/2009WAF2222251.1.
Ebert,E.E.,and W.A.Gallus Jr.,2009:Toward better understanding of the contiguous rain area(CRA)method for spatial forecast verification.Wea.Forecasting,24,1401-1415,doi:10.1175/2009WAF2222252.1.
Gilleland,E.,2016:A new characterization in the spatial verification framework for false alarms,misses,and overall patterns.Wea.Forecasting,32,187-198,doi:10.1175/WAF-D-16-0134.1.
Lorenz,E.N.,1969:Atmospheric predictability as revealed by naturally occurring analogues.J.Atmos.Sci,26,636-646,doi:10.1175/1520-0469(1969)26<636:APARBN>2.0.CO;2.
Mass,C.F.,D.Ovens,K.Westrick,et al.,2002:Does increasing horizontal resolution produce more skillful forecasts?Bull Amer.Meteor.Soc,83,407-430,doi:10.1175/1520-0477(2002)083<0407:DIHRPM>2.3.CO;2.
Mittermaier,M.,and N.Roberts,2010:Intercomparison of spatial forecast verification methods:Identifying skillful spatial scales using the fractions skill score.Wea.Forecasting,25,343-354,doi:10.1175/2009WAF2222260.1.
Murphy,A.H.,1971:A note on the ranked probability score.J.Appl.Meteor.,10,155-156,doi:10.1175/1520-0450(1971)010<0155:ANOTRP>2.0.CO;2.
Murphy,A.H.,1993:What is a good forecast?An essay on the nature of goodness in weather forecasting Wea.Forecasting,8,281-293,doi:10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.
Roberts,N.,2008:Assessing the spatial and temporal variation in the skill of precipitation forecasts from an NWP model.Meteor.Appl.,15,163-169,doi:10.1002/(ISSN)1469-8080.
Roberts,N.,and H.W.Lean,2008:Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events.Mon.Wea.Rev.,136,78-97,doi:10.1175/2007MWR2123.1.
Taylor,K.E.,2001:Summarizing multiple aspects of model performance in a single diagram.J.Georphys.Res.,106,7183-7192,doi:10.1029/2000JD900719.
Weisman,M.L.,C.Davis,W.Wang,et al.,2008:Experiences with 0-36-h explicit convective forecasts with the WRF-ARWmodel.Wea.Forecasting,23,407-437,doi:10.1175/2007WAF2007005.1.
Yates,E.,S.Anquetin,V.Ducrocq,et al.,2006:Point and areal validation of forecast precipitation fields.Meteor.Appl.,13,1-20.
Zepeda-Arce,J.,E.Foufoula-Georgiou,and K.K.Droegemeier,2000:Space-time rainfall organization and its role in validating quantitative precipitation forecasts.J.Geophys.Res.,105,10129-10146,doi:10.1029/1999JD901087.
Zhao,B.,and B.Zhang,2017:Application of neighborhood spatial verification method on precipitation evaluation.Torrential Rain and Disasters,36,497-504,doi:10.3969/j.issn.1004-045.2017.06.004.(in Chinese)
Atger,F.,2001:Verification of intense precipitation forecasts from single models and ensemble prediction systems.Nonlin.Proc.Geophys.,8,401-417,doi:10.5194/npg-8-401-2001.
Baldwin,M.E.,and J.S.Kain,2006:Sensitivity of several performance measures to displacement error,bias,and event frequency.Wea.Forecasting,21,636-648,doi:10.1175/WAF933.1.
Brill,K.F.,and F.Mesinger,2009:Applying a general analytic method for assessing bias sensitivity to bias-adjusted threat and equitable threat scores.Wea.Forecasting,24,1748-1754,doi:10.1175/2009WAF2222272.1.
Casati,B.,2010:New developments of the intensity-scale technique within the Spatial Verification Methods Intercomparison Project.Wea.Forecasting,25,113-143,doi:10.1175/2009WAF2222257.1.
Casati,B.,G.Ross,and D.B.Stephenson,2004:A new intensityscale approach for the verification of spatial precipitation forecasts.Meteor.Appl.,11,141-154,doi:10.1017/S1350482704001239.
Casati,B.,L.J.Wilson,D.B.Stephenson,et al.,2008:Forecast verification:Current status and future directions.Meteor.Appl.,15,3-18,doi:10.1002/(ISSN)1469-8080.
Chen,J.,Y.Wang,L.Li,et al.,2013:A unified verification system for operational models from Regional Meteorological Centres of China Meteorological Administration.Meteor.Appl.,20,140-149,doi:10.1002/met.1406.
Davis,C.A.,B.G.Brown,R.Bullock,et al.,2009:The method for Object-Based Diagnostic Evaluation(MODE)applied to numerical forecasts from the 2005 NSSL/SPC Spring Program.Wea.Forecasting,24,1252-1267,doi:10.1175/2009WAF2222241.1.
Ebert,E.E.,2008:Fuzzy verification of high-resolution gridded forecasts:A review and proposed framework.Meteor.Appl.,15,51-64,doi:10.1002/(ISSN)1469-8080.
Ebert,E.E.,2009:Neighborhood verification:A strategy for rewarding close forecasts.Wea.Forecasting,24,1498-1510,doi:10.1175/2009WAF2222251.1.
Ebert,E.E.,and W.A.Gallus Jr.,2009:Toward better understanding of the contiguous rain area(CRA)method for spatial forecast verification.Wea.Forecasting,24,1401-1415,doi:10.1175/2009WAF2222252.1.
Gilleland,E.,2016:A new characterization in the spatial verification framework for false alarms,misses,and overall patterns.Wea.Forecasting,32,187-198,doi:10.1175/WAF-D-16-0134.1.
Lorenz,E.N.,1969:Atmospheric predictability as revealed by naturally occurring analogues.J.Atmos.Sci,26,636-646,doi:10.1175/1520-0469(1969)26<636:APARBN>2.0.CO;2.
Mass,C.F.,D.Ovens,K.Westrick,et al.,2002:Does increasing horizontal resolution produce more skillful forecasts?Bull Amer.Meteor.Soc,83,407-430,doi:10.1175/1520-0477(2002)083<0407:DIHRPM>2.3.CO;2.
Mittermaier,M.,and N.Roberts,2010:Intercomparison of spatial forecast verification methods:Identifying skillful spatial scales using the fractions skill score.Wea.Forecasting,25,343-354,doi:10.1175/2009WAF2222260.1.
Murphy,A.H.,1971:A note on the ranked probability score.J.Appl.Meteor.,10,155-156,doi:10.1175/1520-0450(1971)010<0155:ANOTRP>2.0.CO;2.
Murphy,A.H.,1993:What is a good forecast?An essay on the nature of goodness in weather forecasting Wea.Forecasting,8,281-293,doi:10.1175/1520-0434(1993)008<0281:WIAGFA>2.0.CO;2.
Roberts,N.,2008:Assessing the spatial and temporal variation in the skill of precipitation forecasts from an NWP model.Meteor.Appl.,15,163-169,doi:10.1002/(ISSN)1469-8080.
Roberts,N.,and H.W.Lean,2008:Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events.Mon.Wea.Rev.,136,78-97,doi:10.1175/2007MWR2123.1.
Taylor,K.E.,2001:Summarizing multiple aspects of model performance in a single diagram.J.Georphys.Res.,106,7183-7192,doi:10.1029/2000JD900719.
Weisman,M.L.,C.Davis,W.Wang,et al.,2008:Experiences with 0-36-h explicit convective forecasts with the WRF-ARWmodel.Wea.Forecasting,23,407-437,doi:10.1175/2007WAF2007005.1.
Yates,E.,S.Anquetin,V.Ducrocq,et al.,2006:Point and areal validation of forecast precipitation fields.Meteor.Appl.,13,1-20.
Zepeda-Arce,J.,E.Foufoula-Georgiou,and K.K.Droegemeier,2000:Space-time rainfall organization and its role in validating quantitative precipitation forecasts.J.Geophys.Res.,105,10129-10146,doi:10.1029/1999JD901087.
Zhao,B.,and B.Zhang,2017:Application of neighborhood spatial verification method on precipitation evaluation.Torrential Rain and Disasters,36,497-504,doi:10.3969/j.issn.1004-045.2017.06.004.(in Chinese)