The relative operating characteristics (ROC) diagram

A powerful way to verify probability forecasts and in particular to compare their perform­ance with deterministic forecast systems, is the two-dimensional “relative operating characteristics” or “ROC” diagram. These categorical forecasts will produce a set of pairs of “hit rate” and “false alarm rate” values to be entered into the ROC diagram: false alarm rate (FR) on the x-axis and hit rate (HR) value on the y-axis. The upper left corner of the ROC diagram represents a perfect forecast sys­tem (no false alarms, only hits). The closer any verification is to this upper left corner, the higher the skill. The lower left corner (no hits or false alarms) represents a system which never warns of an event. The upper right corner represents a system where the event is always warned for (see Figure 89).

Figure 89:  The principle of the ROC diagram: a large number of probability forecasts are turned into categorical forecasts depending on whether the probability values of individual forecasts are above or below a certain threshold. The false alarm rate and the hit rate are calculated, thus determining the position in the diagram (red filled circle).

Probabilistic forecasts are transformed into categorical yes/no forecasts defined by thresholds varying from 0% to 100% (see Figure 90).

Figure 90:  The same as above, but repeated for several thresholds between 0 and 100%, including the hit rates and false alarm rates of the deterministic model; although not providing probabilistic predictions it can be represented on the diagram by its typical hit rate and false alarm rate (green filled circle).

The ROC score is the area underneath the forecast curve (see Figure 91).

Figure 91:  The area underneath the points, joined by straight lines, defines the ROC area, which is, ideally, 1.0 and at worst 0.0. Random forecasts yield 0.5, the triangular area underneath the 45° line.

There are two schools on how to calculate this: either with a smooth spline or linearly, connecting the points.