Kappa Statistic. The Kappa value for each pair of classifiers is presented in Table 5. The Kappa analysis indicates low and inadequate agreements in general according to the interpretation in Table 3. However, no negative values are achieved; there is only one pair that agrees moderately and none with good or very good agreement, i.e. achieves Kappa values above or equal to 0.41. The average Kappa value is 0.16, which is considered as a very low overall
Kappa Statistic. 15 The kappa statistic is a skill score which measures how well an analyst can perform compared to chance. In forecast verification the kappa statistics in known as the Heidke Skill Score, and it is the skill score constructed from the percent correct against random chance (Xxxxx, 2011; Xxxxxxx and Xxxxxxxxxx, 2012). The kappa statistic can be calculated for any contingency table to measure the level of agreement between analysts and the segmentation algorithm. This measure takes into account the possibility of chance agreement between analysts and MAGIC when determining the agreement found between them. 20 The kappa statistic, κ, is calculated as κ = p0 − pe Σ 1 − pe
Kappa Statistic. After collecting all of the marking results from all of the expert raters, Xxxxx’x j (kappa) statistic was calculated for each pair of raters in order to better observe the distribution of IRA. It is calcu- lated as: ¼ 1 Pr e j PrðaÞ— PrðeÞ — ð Þ where Xx(a) is the relative observed agreement between raters, and Pr(e) is the hypothetical probability of chance agreement, using the observed responses to calculate the probabilities of each observer randomly assigned to each category. Kappa has been described as the ideal statistic to quantify agreement for dichotomous variables. Magnitude guidelines in the literature suggested that: values <0 as indicating no agreement, 0–0.20 as slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect agreement (Xxxxxx and Xxxx, 1977). Implicit in the kappa is the assumption that the rated items, subjects, or targets are independent. However, identification of ‘‘transient events’’ during a serially observed process such as sei- zures in EEG data contains responses that are highly correlated among the neighboring responses, which violates the indepen- dence assumption of kappa. Therefore, in this study, we applied a Xxxxx-Xxxxx-based permutation technique to produce an empiri- cal distribution of kappa in the presence of dependence (Xxxxxx and Xxxxx, 2007). The main purpose of this technique is to calculate expected agreement due to chance (i.e., Pr(e)) between two raters. To achieve this, we first generated two sequences (one for seizure events and the other for PDs) comprised of binary responses from each rater’s markings. Each binary response represents the mark- ing in each second – i.e., 1 if the second is within an event marking and 0, otherwise. Secondly, for each binary sequence, 10,000 ran- dom permutations of runs of 1 s and 0 s were sampled, and the pairs of permuted sequences were cross-tabulated to create an agreement table. Repetition of this permutation process provided a sample from all possible random agreements of all possible pairs of sequences. The R statistics and development system was used to perform the simulations.
Kappa Statistic. The results of the estimation of the kappa statistic for the Math-In-Use suggest agreement among raters in interpreting participants’ discussion of mathematics in the scenario was good, κ = 0.6745, p ≤ .001. The degree to which raters were able to independently classify responses at the lower end of the distribution (Category A, No Math) was excellent, κ = 0.8607, p ≤ .
