If the data do have an order, the information in the measurements is not fully taken advantage of.Ĭorrelation coefficients Main articles: Pearson product-moment correlation coefficient, Spearman's rank correlation coefficientĮither Pearson's r ,Īs it takes into account of the differences in ratings for individual segments, along with the correlation between raters. They suffer from the same problem as the joint-probability in that they treat the data as nominal and assume the ratings have no natural ordering. Kappa statistics Main articles: Cohen's kappa, Fleiss' kappaĬohen's kappa, which works for two raters, and Fleiss' kappa, an adaptation that works for any fixed number of raters, improve upon the joint probability in that they take into account the amount of agreement that could be expected to occur through chance. It does not take into account that agreement may happen solely based on chance. It assumes that the data are entirely nominal. 5) is assigned by each rater and then divides this number by the total number of ratings. It is the number of times each rating (e.g. The joint-probability of agreement is probably the most simple and least robust measure. Some options are: joint-probability of agreement, Cohen's kappa and the related Fleiss' kappa, inter-rater correlation, concordance correlation coefficient and intra-class correlation. Different statistics are appropriate for different types of measurement. There are a number of statistics which can be used to determine inter-rater reliability. If various raters do not agree, either the scale is defective or the raters need to be re-trained. It is useful in refining the tools given to human judges, for example by determining if a particular scale is appropriate for measuring a particular variable. It gives a score of how much homogeneity, or consensus, there is in the ratings given by judges. Inter-rater reliability, inter-rater agreement, or concordance is the degree of agreement among raters.

Inter rater reliability psychology