The Cohen's Kappa is a statistic that is used to measure inter-rater reliability (and also Intra-rater reliability) for qualitative (categorical) items. It is generally thought to be a more robust measure than simple percent agreement calculation, as Cohen’s Kappa takes into account the possibility of the agreement occurring by chance.
Simple Cohen’s Kappa is calculated as:
κ = (Po - Pe) / (1 - Pe)
Po is the observed agreement among raters, and
Pe is the hypothetical probability of chance agreement.
The Kappa statistic (κ) is a number between -1 and 1. The maximum value means complete agreement; zero or lower means chance agreement.
In the context of machine learning, it's used as a metric to compare the performance of different models. For binary classification problems, it's a valuable metric, especially when the classes are imbalanced.
However, we should clarify that there isn't a metric known as "Weightless Cohen Kappa". There is a version called "Weighted Kappa", also known as the quadratic weighted kappa or Cohen’s kappa. This version is an extension of Cohen's Kappa for ordinal categorical data. It considers the order of the categories and weights disagreements differently (disagreements at further distance are weighted more heavily).
But for binary classification and non-ordinal multi-class classification, we typically use the simple (unweighted) Cohen's Kappa.
Updated 5 months ago