Statistical Testing for the Metrics of Averages. (i.e., metrics of intervals, such as mean time to repair or average delay days): The one-tailed permutation test is used to evaluate performance on the Average parity metrics – metrics for which performance is measured as an “average” (also referred to here as a “mean.”) Its general logic is as follows: Combine the ILEC and CLEC observations into one group, where the total number of observations is nilec+ nclec. Take a sufficiently large number of random samples of size nclec (e.g., 500,000). Record the mean of each re-sample of size nclec. Sort the re-sampled means from best to worst (left to right) and compare where on the distribution of re-sampled means the original CLEC mean is located. If 5% or less of the means lie to the right of the reported CLEC mean, then reject the null hypothesis that the original CLEC sample and the original ILEC sample came from the same population. If the null hypothesis is correct, a permutation test yields a probability value (p value) representing the probability that the difference (or larger) in the ILEC and CLEC sample means is due to random variation. Permutation test’s p values are transformed into “Z score equivalents.” These "Z score equivalents" refer to the standard normal Z score that has the same probability as the p-values from the permutation test. Specifically, this statistical score equivalent refers to the inverse of the standard normal cumulative distribution associated with the probability of seeing the reported CLEC mean, or worse, in the distribution of re-sampled permutation test means. A Z score of less than or equal to –1.645 occurs at most 5% of the time under the null hypothesis that the CLEC mean is at least equal to or better than the ILEC mean. A Z score greater than –1.645 (p-value greater than 5%) supports the belief that the CLEC mean is at least equal to or better than the ILEC mean. For reporting purposes, Z score equivalents equal to or greater than 5.0000 are displayed on monthly reports as 5.0000. Similarly, values for Z statistics equal to or less than –5.0000 are displayed as –5.0000. The following steps are involved in the permutation test procedure:
