Or why you shouldn’t belief predict_proba strategies
In earlier articles, I identified the significance of understanding how certain a mannequin is about its predictions.
For classification issues, it isn’t useful to solely know the ultimate class. We want extra info to make well-informed choices in downstream processes. A classification mannequin that solely outputs the ultimate class covers vital info. We have no idea how certain the mannequin is and the way a lot we are able to belief its prediction.
How can we obtain extra belief within the mannequin?
Two approaches can provide us extra perception into classification issues.
We may flip our level prediction right into a prediction set. The purpose of the prediction set is to ensure that it comprises the true class with a given chance. The dimensions of the prediction set then tells us how certain our mannequin is about its prediction. The less courses the prediction set comprises, the surer the mannequin is.