Receiver operating characteristic (ROC) curve
Many classifiers can be made to work off a threshold. For example, in anomaly detection, if the anomaly scores computed by the predictive model for certain observations cross a certain threshold, then we may deem those observations to be anomalous.
In the context of such threshold-based classifiers, a single precision/recall number may not provide the complete performance profile.
Precision and recall are computed for a particular assignment of examples into positive and negative classes. By changing the threshold, the assignments will change.
The receiver operating characteristic (ROC) curve allows the evaluation of such a complete performance profile of the model.
ROC curve is the plot of true positive rate (recall) versus the true negative rate (specificity) for changing values of the threshold.
We compute the recall and specificity for each value of threshold between the maximum and minimum possible value for the thresholds.
This exercise provides a list of paired recall and specificity values, which when plotted result in the ROC curve.
For the highest value of threshold, all examples are classified as negative. So the true negative rate is 1.0 and the true positive rate is 0.0.
Conversely, for the lowest value of threshold, all examples are classified as positive. So the true negative rate is 0.0 and true positive rate is 1.0.
If the coordinates are indicated as tuples of paired values \( (\text{true positive rate}, \text{true negative rate}) \), then, the diagonal line from \( (0,0) \) to \( (1,1) \) will be the ROC curve of a random model — one that assigns the scores randomly to observations.
A good classifier should then have an ROC curve above this random line, to imply learning capability that is better than random categorization.
Among multiple good classifiers, the one with the ROC curve that passes closest to the coordinate \( (1,0) \) will be the winner.
In addition to being an evaluation metric, the ROC curve is also used to discover optimal thresholds. If giving equal weighting to recall and specificity, the point along the ROC curve that is closest to the coordinate \( (1,0) \) is considered as a suitable threshold.