False systematic errors

One of the complexities of interpreting the ME is that apparent systematic errors might, in fact, have a non-systematic origin. If this is the case, a perfect model appears to have systematic errors; a stable model appears to suffer from model drift. This is a reflection of a general statistical artefact, the “regression to the mean” effect [1] .

The fact that a perfect model forecasts anomalies with the same intensity and frequencies as observed does not mean that they will be correct in time and place. Due to decreasing predictive skill it will, with increasing lead time, have less success in getting the forecast anomalies right in intensity, time and place. If the forecast is wrong for a specific forecast anomaly, in particular for a strong anomaly,  the verifying truth might be more anomalous but in most cases will be less anomalous. Even if the forecast anomaly has the right intensity, phase errors will tend to displace it rather towards less anomalous patterns than towards even more anomalous configurations (see Figure 62).

Figure 62:  A schematic picture of a medium range forecast (black) and the verifying analysis. The forecast anomalies have about the same magnitudes as the verifying anomalies, but they are out of phase. This will yield a tendency for positive anomalies to verify against less positive or even negative anomalies, negative anomalies to verify against less negative or even positive anomalies.

Anomalies will therefore appear as if they have been systematically exaggerated, increasingly so as skill decreases with increasing lead time. Plotted in a scatter diagram, these non-systematic forecast errors therefore give a misleading impression that positive anomalies are systematically over-forecast and negative anomalies systematically under-forecast (see Figure 63).

Figure 63: A scatter diagram of forecasts versus forecast error. Warm forecasts appear too warm, cold forecasts appear too cold. If the forecasts are short range, it is reasonable to infer that the system is over-active, overdeveloping warm and cold anomalies. If, on the other hand, the forecasts are well into the medium range, this might not be the case. Due to decreased forecast skill, predicted anomalies tend to verify against less anomalous observed states.