| Abstract: |
| The successful EnKF prediction skill with an ensemble size K much smaller than the dimension d is an intriguing mystery. The practitioners often attribute this success to a low effective dimension p, of which the formal definition is unclear. The first part of our framework proposes a natural definition for the effective dimension, using the covariance spectrum of an associated Kalman filter. The second component employs the Mahalanobis norm to quantify the EnKF performance, which is intrinsically dissipative for Kalman type of filter updates. This dissipative mechanism is stable enough to wither the noisy perturbation from model or small sampling error. The low effective dimension plays a vital role here, since when K > Cp for a constant C, the sample forecast covariance matrix can concentrate around its expected value, using a new random matrix theory result. Practical covariance inflation and spectral projection are employed to our EnKF. The fact that these augmentations are necessary for our proof, indicates the theoretical significance of these augmentations, while their practical significance has already been observed and well documented. |
|