# Low Dimensionality in the Atmospheric Dynamics: Implications for Data Assimilation

Friday, September 28, 2001 - 2:00pm - 3:00pm

Keller 3-180

Eugenia Kalnay (University of Maryland)

Joint work with Matteo Corazza, DJ Patil, Rebecca Morss, Brian Hunt, Ed Ott, Ming Cai and Jim Yorke (University of Maryland, College Park, MD 20742-2425).

We introduced a statistic, the BV-dimension, to measure the effective local finite-time dimensionality of the atmosphere. We show that this dimension is often quite low, and suggest that this finding has important implications for data assimilation and the accuracy of weather forecasting (Patil et al, 2001).

The original database for this study was the forecasts of the NCEP global ensemble forecasting system. The initial differences between the control forecast and the perturbed forecasts are called bred vectors. The control and perturbed initial conditions valid at time t=nDt are evolved using the forecast model until time t=(n+1) Dt. The differences between the perturbed and the control forecasts are scaled down to their initial amplitude, and constitute the bred vectors valid at (n+1) Dt. Their growth rate is typically about 1.5/day. The bred vectors are similar by construction to leading Lyapunov vectors except that they have small but finite amplitude, and they are valid at finite times.

The original NCEP ensemble data set has 5 independent bred vectors. We define a local bred vector at each grid point by choosing the 5 by 5 grid points centered at the grid point (a region of about 110km by 1100km), and using the north-south and east-west velocity components at 500mb pressure level to form a 50 dimensional column vector. Since we have k=5 global bred vectors, we also have k local bred vectors at each grid point. We estimate the effective dimensionality of the subspace spanned by the local bred vectors by performing a singular value decomposition (EOF analysis). The k local bred vector columns form a 50xk matrix M. The singular values [IMAGE] of M measure the extent to which the k column unit vectors making up the matrix M point in the direction of [IMAGE]. We define the bred vector dimension as[IMAGE]. For example, if 4 out of the 5 vectors lie along [IMAGE], and one lies along[IMAGE], the BV-dimension would be [IMAGE], less than 2 because one direction is more dominant than the other in representing the original data.

The results (Patil et al, 2001) show that there are large regions where the bred vectors span a subspace of substantially lower dimension than that of the full space. These low dimensionality regions are dominant in the baroclinic extratropics, typically have a lifetime of 3-7 days, have a well-defined horizontal and vertical structure that spans most of the atmosphere, and tend to move eastward (Fig.1). New results with a large number of ensemble members confirm these results and indicate that the low dimensionality regions are quite robust, and depend only on the verification time (i.e., the underlying flow). Corazza et al (2001) have performed experiments with a data assimilation system based on a quasi-geostrophic model and simulated observations (Morss, 1999, Hamill et al, 2000). A 3D-variational data assimilation scheme for a quasi-geostrophic channel model is used to study the structure of the background error and its relationship to the corresponding bred vectors. The ^Ótrue^Ô evolution of the model atmosphere is defined by an integration of the model and ^Órawinsonde observations^Ô are simulated by randomly perturbing the true state at fixed locations.

It is found that after 3-5 days the bred vectors develop well organized structures which are very similar for the two different norms considered in this paper (potential vorticity norm and streamfunction norm). The results show that the bred vectors do indeed represent well the characteristics of the data assimilation forecast errors, and that the subspace of bred vectors contains most of the forecast error, except in areas where the forecast errors are small. For example, the angle between the 6hr forecast error and the subspace spanned by 10 bred vectors is less than 10o over 90% of the domain, indicating a pattern correlation of more than 98.5% between the forecast error and its projection onto the bred vector subspace.

Case studies using different observational densities are considered to compare the evolution of the Bred Vectors to the spatial structure of the background error. Bred vectors obtained using the ^Ótrue atmosphere^Ô (which would not be possible in an operational center) and analysis are very similar, even when using a low density observing network. This indicates that the bred vectors (and by inference the forecast errors) are more likely dependent on the large scale characteristics of the flow, which are usually captured in an analysis.

In addition, the bred vector dimension (BV-dimension), defined by Patil et al., (2001) is applied to the bred vectors. It is found that the local dimension is usually much smaller (between 2 and 4) than the number of bred vectors, particularly in those areas where the errors are large.

The presence of low-dimensional regions in the perturbations of the basic flow has important implications for data assimilation. At any given time, there is a difference between the true atmospheric state and the model forecast. Assuming that model errors are not the dominant source of errors, in a region of low BV-dimensionality the difference between the true state and the forecast should lie substantially in the low dimensional unstable subspace of the few bred vectors that contribute most strongly to the low BV-dimension. This information should yield a substantial improvement in the forecast: the data assimilation algorithm should correct the model state by moving it closer to the observations along the unstable subspace, since this is where the true state most likely lies.

This can be seen in a simple example based on the 3-dimensional Variational data assimilation (3D-Var) formulation. If we assume that observations [IMAGE] have a diagonal error covariance [IMAGE], and if the local unstable subspace is spanned by [IMAGE], we can assume that locally the background error covariance is of the form [IMAGE] Then the minimum of the cost function for 3D-Var [IMAGE] is attained for the analysis given by [IMAGE] (Kalnay and Toth, 1994).

Note that this involves just scalar products and shows that the correction takes place along the bred vector subspace. The local bred vectors in low dimensionality regions give a representation of the ^Óerrors of the day^Ô in the data assimilation, which depend on the evolving underlying flow.

Preliminary experiments have been conducted with the quasi-geostrophic data assimilation system testing whether it is possible to add ^Óerrors of the day^Ô based on bred vectors to the standard (constant) 3D-Var background error covariance in order to capture these important errors. The results are extremely encouraging, indicating a significant reduction in the analysis errors at a very low computational cost (Figs. 2 and 3).

References:

Corazza, M., E. Kalnay, DJ Patil, R. Morss, M Cai, I. Szunyogh, BR Hunt, E Ott and JA Yorke, 2001: Use of the breeding technique to estimate the structure of the analysis ^Óerrors of the day^Ô. Submitted to Nonlinear Processes in Geophysics.

Hamill, T.M., Snyder, C., and Morss, R.E., 2000: A Comparison of Probabilistic Forecasts from Bred, Singular-Vector and Perturbed Observation Ensembles, Mon. Wea. Rev., 128, 1835--1851. Kalnay, E., and Z. Toth, 1994: Removing growing errors in the analysis cycle. Preprints of the Tenth Conference on Numerical Weather Prediction, Amer. Meteor. Soc., 1994, 212-215.

Morss, R. E., 1999: Adaptive observations: Idealized sampling strategies for improving numerical weather prediction. PHD thesis, Massachussetts Institute of technology, 225pp. Patil, D. J. S., B. R. Hunt, E. Kalnay, J. A. Yorke, and E. Ott., 2001: Local Low Dimensionality of Atmospheric Dynamics. Phys. Rev. Lett., 86, 5878.

Fig. 1: Example of the 3-dimensional effective dimension (BVD) of the bred vectors corresponding to 20 March 2000. Blue colors represent a local BVD of about 5 (the number of bred vectors). Red represents a local effective dimensionality close to 1. The vertical slices are computed independently from each other.

[IMAGE]

[IMAGE]

Fig. 2: Example of a year of analysis errors based on the regular 3D-Var data assimilation with an optized constant background error covariance (black, average yellow). It shows strong ^Óerrors of the day^Ô that are not captured by the standard methods. The green analysis errors were obtained by augmenting the constant background error covariance with a sum of 10 bbT, where b are the global bred vectors of the day.

We introduced a statistic, the BV-dimension, to measure the effective local finite-time dimensionality of the atmosphere. We show that this dimension is often quite low, and suggest that this finding has important implications for data assimilation and the accuracy of weather forecasting (Patil et al, 2001).

The original database for this study was the forecasts of the NCEP global ensemble forecasting system. The initial differences between the control forecast and the perturbed forecasts are called bred vectors. The control and perturbed initial conditions valid at time t=nDt are evolved using the forecast model until time t=(n+1) Dt. The differences between the perturbed and the control forecasts are scaled down to their initial amplitude, and constitute the bred vectors valid at (n+1) Dt. Their growth rate is typically about 1.5/day. The bred vectors are similar by construction to leading Lyapunov vectors except that they have small but finite amplitude, and they are valid at finite times.

The original NCEP ensemble data set has 5 independent bred vectors. We define a local bred vector at each grid point by choosing the 5 by 5 grid points centered at the grid point (a region of about 110km by 1100km), and using the north-south and east-west velocity components at 500mb pressure level to form a 50 dimensional column vector. Since we have k=5 global bred vectors, we also have k local bred vectors at each grid point. We estimate the effective dimensionality of the subspace spanned by the local bred vectors by performing a singular value decomposition (EOF analysis). The k local bred vector columns form a 50xk matrix M. The singular values [IMAGE] of M measure the extent to which the k column unit vectors making up the matrix M point in the direction of [IMAGE]. We define the bred vector dimension as[IMAGE]. For example, if 4 out of the 5 vectors lie along [IMAGE], and one lies along[IMAGE], the BV-dimension would be [IMAGE], less than 2 because one direction is more dominant than the other in representing the original data.

The results (Patil et al, 2001) show that there are large regions where the bred vectors span a subspace of substantially lower dimension than that of the full space. These low dimensionality regions are dominant in the baroclinic extratropics, typically have a lifetime of 3-7 days, have a well-defined horizontal and vertical structure that spans most of the atmosphere, and tend to move eastward (Fig.1). New results with a large number of ensemble members confirm these results and indicate that the low dimensionality regions are quite robust, and depend only on the verification time (i.e., the underlying flow). Corazza et al (2001) have performed experiments with a data assimilation system based on a quasi-geostrophic model and simulated observations (Morss, 1999, Hamill et al, 2000). A 3D-variational data assimilation scheme for a quasi-geostrophic channel model is used to study the structure of the background error and its relationship to the corresponding bred vectors. The ^Ótrue^Ô evolution of the model atmosphere is defined by an integration of the model and ^Órawinsonde observations^Ô are simulated by randomly perturbing the true state at fixed locations.

It is found that after 3-5 days the bred vectors develop well organized structures which are very similar for the two different norms considered in this paper (potential vorticity norm and streamfunction norm). The results show that the bred vectors do indeed represent well the characteristics of the data assimilation forecast errors, and that the subspace of bred vectors contains most of the forecast error, except in areas where the forecast errors are small. For example, the angle between the 6hr forecast error and the subspace spanned by 10 bred vectors is less than 10o over 90% of the domain, indicating a pattern correlation of more than 98.5% between the forecast error and its projection onto the bred vector subspace.

Case studies using different observational densities are considered to compare the evolution of the Bred Vectors to the spatial structure of the background error. Bred vectors obtained using the ^Ótrue atmosphere^Ô (which would not be possible in an operational center) and analysis are very similar, even when using a low density observing network. This indicates that the bred vectors (and by inference the forecast errors) are more likely dependent on the large scale characteristics of the flow, which are usually captured in an analysis.

In addition, the bred vector dimension (BV-dimension), defined by Patil et al., (2001) is applied to the bred vectors. It is found that the local dimension is usually much smaller (between 2 and 4) than the number of bred vectors, particularly in those areas where the errors are large.

The presence of low-dimensional regions in the perturbations of the basic flow has important implications for data assimilation. At any given time, there is a difference between the true atmospheric state and the model forecast. Assuming that model errors are not the dominant source of errors, in a region of low BV-dimensionality the difference between the true state and the forecast should lie substantially in the low dimensional unstable subspace of the few bred vectors that contribute most strongly to the low BV-dimension. This information should yield a substantial improvement in the forecast: the data assimilation algorithm should correct the model state by moving it closer to the observations along the unstable subspace, since this is where the true state most likely lies.

This can be seen in a simple example based on the 3-dimensional Variational data assimilation (3D-Var) formulation. If we assume that observations [IMAGE] have a diagonal error covariance [IMAGE], and if the local unstable subspace is spanned by [IMAGE], we can assume that locally the background error covariance is of the form [IMAGE] Then the minimum of the cost function for 3D-Var [IMAGE] is attained for the analysis given by [IMAGE] (Kalnay and Toth, 1994).

Note that this involves just scalar products and shows that the correction takes place along the bred vector subspace. The local bred vectors in low dimensionality regions give a representation of the ^Óerrors of the day^Ô in the data assimilation, which depend on the evolving underlying flow.

Preliminary experiments have been conducted with the quasi-geostrophic data assimilation system testing whether it is possible to add ^Óerrors of the day^Ô based on bred vectors to the standard (constant) 3D-Var background error covariance in order to capture these important errors. The results are extremely encouraging, indicating a significant reduction in the analysis errors at a very low computational cost (Figs. 2 and 3).

References:

Corazza, M., E. Kalnay, DJ Patil, R. Morss, M Cai, I. Szunyogh, BR Hunt, E Ott and JA Yorke, 2001: Use of the breeding technique to estimate the structure of the analysis ^Óerrors of the day^Ô. Submitted to Nonlinear Processes in Geophysics.

Hamill, T.M., Snyder, C., and Morss, R.E., 2000: A Comparison of Probabilistic Forecasts from Bred, Singular-Vector and Perturbed Observation Ensembles, Mon. Wea. Rev., 128, 1835--1851. Kalnay, E., and Z. Toth, 1994: Removing growing errors in the analysis cycle. Preprints of the Tenth Conference on Numerical Weather Prediction, Amer. Meteor. Soc., 1994, 212-215.

Morss, R. E., 1999: Adaptive observations: Idealized sampling strategies for improving numerical weather prediction. PHD thesis, Massachussetts Institute of technology, 225pp. Patil, D. J. S., B. R. Hunt, E. Kalnay, J. A. Yorke, and E. Ott., 2001: Local Low Dimensionality of Atmospheric Dynamics. Phys. Rev. Lett., 86, 5878.

Fig. 1: Example of the 3-dimensional effective dimension (BVD) of the bred vectors corresponding to 20 March 2000. Blue colors represent a local BVD of about 5 (the number of bred vectors). Red represents a local effective dimensionality close to 1. The vertical slices are computed independently from each other.

[IMAGE]

[IMAGE]

Fig. 2: Example of a year of analysis errors based on the regular 3D-Var data assimilation with an optized constant background error covariance (black, average yellow). It shows strong ^Óerrors of the day^Ô that are not captured by the standard methods. The green analysis errors were obtained by augmenting the constant background error covariance with a sum of 10 bbT, where b are the global bred vectors of the day.