An answering system can decline questions when its evidence is insufficient. Selective answering measures both how often the system answers and how often those answers are wrong.
Use two denominators
Coverage is answered questions divided by all questions. Error risk is incorrect answers divided by answered questions. Abstentions are not counted as correct answers.
In the card, answering all 100 questions produces 10 wrong answers: coverage is 100% and risk is 10%. A selective policy answers 60 questions and gets three wrong: coverage is 60% and risk is 5%. It makes fewer errors among its answers but leaves 40 questions unanswered.
Choose a threshold with held-out data
Varying an abstention threshold gives a risk-coverage curve. Evaluate the threshold on held-out questions that resemble the intended use. A model’s confident wording is not enough to establish that its answer is reliable.
The acceptable balance depends on the cost of an incorrect answer and the cost of declining one. Distribution shifts can change the curve after deployment. Monitor both quantities so that an apparent accuracy improvement does not hide a system that has stopped answering useful questions.