The difference between the accuracy of real and the corresponding random model is a useful parameter for validation of two-state classification model quality

Batista, Jadranko; Vikić-Topić, Dražen; Lučić, Bono (2016) The difference between the accuracy of real and the corresponding random model is a useful parameter for validation of two-state classification model quality. Croatica chemica acta, 89 (4). pp. 527-534. ISSN 0011-1643

Preview

PDF - Published Version - article Available under License Creative Commons Attribution. Download (630kB) | Preview

Abstract

The simplest and the most commonly used measure for assess the classification model quality is parameter Q2 = 100 (p + n) / N (%) named the classification accuracy, p, n and N are the total numbers of correctly predicted compounds in the first and in the second class, and the total number of elements of classes (compounds) in data set, respectively. Moreover, the most probable accuracy that can be obtained by a random model is calculated for two-state model by the formulae Q2, rnd = 100 [(p + u) (p + o) + (n + u) (n + o)] / N2 (%), where u and o are the total number of under- predictions (when class 1 is predicted by the model as class 2) and over-predictions (when class 2 is predicted by the model as class 1) in data set, respectively. Finally, the difference between these two parameter ΔQ2 = Q2 – Q2, rnd is introduced, and it is suggested to compute and give ΔQ2 for each two-state classification model to assess its contribution over the accuracy of the corresponding random model. When data set is ideally balanced having the same numbers of elements in both classes, the two-state classification problem is the most difficult with maximal Q2 = 100 % and Q2, rnd = 50 %, giving the maximal ΔQ2 = 50 %. The usefulness of ΔQ2 parameter is illustrated in comparative analysis on two-class classification models from literature for prediction of secondary structure of membrane proteins and on several quanti¬tative structure- property models. Real contributions of these models over the random level of accuracy is calculated, and their ΔQ2 values are compared mutually and with the value of ΔQ2 (= 50 %) for the most difficult two-state classification model.

Item Type:	Article
Uncontrolled Keywords:	classification model ; Q2 accuracy ; overall classification accuracy ; random classification accuracy ; classification accuracy difference ; correct class estimation ; under-prediction ; over-prediction ; class imbalance ; membrane structure modeling ; QSAR classification modeling
Subjects:	NATURAL SCIENCES > Physics
Divisions:	NMR Center
Depositing User:	Bono Lučić
Date Deposited:	04 Nov 2020 11:37
URI:	http://fulir.irb.hr/id/eprint/5972
DOI:	10.5562/cca3117

Actions (login required)

View Item