The Costs and Potential Benefits of Introducing the “I Don’t Know” Answer in Binary Classification Settings

We are of the opinion that during the design of a binary classifier one ought to consider adding an “I don’t know” answer. We provide the case for the introduction of this third category when a human needs to make a decision based on the answer from a binary classifier. We discuss the costs and potential benefits of its introduction. Colloquially, we have used the term “I don’t know”, but formally we refer to it as NotAvailable. A procedure to define NotAvailable predictions in binary classifiers, called all leave-one-out models (ALOOM), is presented as proof of the concept. Furthermore, we discuss the potential benefits of applying ALOOM in real life applications.