The problem is to determine whether a patient referred to the clinic is hypothyroid. Therefore three classes are built: normal (not hypothyroid), hyperfunction and subnormal functioning. Because 92 percent of the patients are not hyperthyroid a good classifier must be significant better than 92%.
This is for some data I found for Thyroid Disease.
Imagine that you are the boss and hire a data scientist at a high salary to build a model that you hope will achieve a high accuracy score.$^{\dagger}$ She comes back to you reporting $92\%$ accuracy. “Wow,” you think. “That sounds like an A in school! Great job you’re worth every penny of your salary!”
Then you realize that you could have gotten that same $92\%$ accuracy by predicting the dominant outcome every time. At this point, you realize that your data scientist has not accomplished anything beyond what you could have accomplished, and you no longer feel that she is worth the high salary you pay her.
In your case, since you know that $92\%$ of the so patients lack the hyperthyroid, you could get a classifier with $92\%$ accuracy just by predicting this every time.
An issue that might get you is that there are three categories, yet the task is to classify into hyperthyroid or not, which is binary. Whether or not these categories should be combined or dealt with separately really warrants a distinct question, however.
$^{\dagger}$A critical aspect of this is the assumption of accuracy being the metric of interest. I am not sold on this. First, any threshold-based metric is known to have issues. Second, even if you are in a position where you must make discrete classifications instead of predicting tendencies like the link discusses, the costs of wrong decisions need not be equal. I find that to be highly plausible here, and you might be willing to sacrifice a bit of accuracy for gains in sensitivity or specificity. The idea of comparing to baseline still applies to such a performance metric, though. If your data scientist cannot make a model that performs better than your naïve model that classifies everyone the same way, she probably hasn’t helped you that much, no matter how much the accuracy score looks like an A in school.
a good classifier must be significant better than 92%.
There are two issues here
Where does this statement come from?
It is not so clear what is meant with the 92%, but it seems to relate to accuracy. $$ \text{accuracy} = \frac{\text{good estimates}}{\text{total estimates}} $$ this 92% accuracy can already by achieved with a naive classifier that assigns every patient to the negative category. For the 92% of the patients that are negative it will be right and for the 8% of the patients that are positive it will be wrong.
But also
Is this statement right?
I disagree. The classifier does not need to be better than 92%.
Imagine the following situation:
Your test could be 100% sensitive and classify 8% of the people as having hyperthyroid disease while they also have the disease, but it also accidentally classifies 9% of the people as having hyperthyroid disease while they actually do not have hyperthyroid disease. Then you make a mistake in 9% of the cases and the accuracy is 91%, which is less than 92%.
Did you do bad?
Since 92% of patients are all from one class, you can build a classifier that just predicts the majority class for all samples and get 92% accuracy. This is of course not a terribly interesting classifier, as it completely ignores everything about individual patients, and makes a prediction based on the population as a whole. Classifiers are generally expected to make a sample-level prediction based on some set of input features, but a naive majority-class classifier can achieve 92% accuracy in this setting without knowing anything at all about individual patients. In some cases where accuracy is the right measure to optimize, the majority-class classifier may be a useful baseline comparator, since it represents an accuracy benchmark that can be achieved with the simplest possible model.
That said, I disagree that any useful classifier must have greater accuracy than the majority-class prevalence. Accuracy is often not the best measure of classifier performance, particularly in cases where the "cost" of errors is not equal between false positives and false negatives. Accuracy treats all misclassification errors as the same, but there can be very real differences in the consequences of making false positive versus false negative errors.
Imagine we are building a classifier for a medical screening test for a serious, fast-progressing, but relatively rare disease. In such a case we want a test with very high sensitivity (we can't afford to miss patients who truly have the disease, since they will die if the disease continues untreated), but are willing to sacrifice specificity to achieve it (we're willing to falsely diagnose some people and send them for a follow-up test). In such a case, we want very high accuracy in the small disease-positive population (resulting in few false negatives), but may allow lower accuracy in the large disease-negative population (resulting in somewhat more false positives). The net accuracy may be lower than the majority-class prevalence, but the overall "cost" of misclassifications is actually lower than other classifiers with higher accuracy, since not all errors are equivalent.
