Threats to Validity. A number of potential limitations in the form of threats to the validity of our study were considered. These were threats to internal and external validity. In this study we focused on evaluating the reliability of assessments based on the SPICE framework. We implicitly assumed that reliability is only a function of the SPICE documents and architecture (e.g., the clarity of practice definitions, the soundness of the rating scheme, and the applicability of the two-dimensional architecture). Threats to internal validity would question this assumtpion. One potential threat to internal validity is a maturation effect. In this study, a maturation effect would be indicated by a change in interrater agreement (as measured by Kappa) over the course of the assessment. For example, as the assessment progresses, assessors may become more fatigued and pay less attention to observing evidence and in making their ratings. This would tend to decrease the extent of interrater agreement as the assessment progresses. Conversely, assessors may gain knowledge of the organization and the way it implements its practices as time progresses. As more evidence is gathered by assessors they may start to converge in their perceptions about the capability of the organization’s processes. This could lead to an increase in interrater agreement as the assessment progresses. If we find a maturation effect then the values of Kappa that we obtained are also a function of when ratings are made during an assessment. To determine if there was a maturation effect, we conducted a number of post-hoc tests. The assessment ratings were made over a 2.5 day period (the whole assessment was longer since it included an initial meeting with management and a closing session where findings were presented). Evidence on nine processes was inspected and ratings were made in the first 1.5 days of the assessment. These were classified as early processes. The remaining six processes were rated in the final day. These were classified as late processes. We tested for differences in the values of Kappa between these two groups. We used a two-tailed test at an alpha level of 0.1. The statistic we used was the Xxxx-Xxxxxxx U test [22]. No diferences were found, and hence there is no evidence that the median Kappa values between the two groups differed. Three different external assessors and five different internal assessors took part in the assessment. The distribution of assessors over time was not uniform, and the...
Threats to Validity. As mentioned earlier, our survey was conducted with only one group of students taking a particular course of a particular university. However, we also mentioned that our sur- vey participants were final-year undergraduate students majoring in computer science who can be considered entry-level developers. Our survey fulfilled its purpose of gauging initial response to our change contract language; our students easily learned and used our change contract language. However, given the number of participants, a larger-scale study is necessary to confirm our results. In particular, more sophisticated study is required to see the validity of several interesting initial observations such as higher correctness rates in structural changes than in behavioral changes and little difference between the correctness rates for artificial programs and real-life programs.
Threats to Validity. Due to the randomness of Randoop, the numbers in third to last columns of Table IV can be varied each time an experiment is performed, although in our experience the gap was not significant. In addition, these numbers are also affected by the limitations of our tool. For example, we found that XMLUnit, a tool we used to check the isomorphism between inputs, occasionally categorized isomorphic inputs as non-isomorphic due to the order sensitiveness of the tool in comparing object graphs. Lastly, our experimental results are confined to a single subject Ant, and we need to conduct experiments with more subjects to generalize the results we obtained to other cases.
Threats to Validity. Although we evaluated the model and layout according to a rubric, they are still graded manually which introduces a bias. On one hand we tried to reduce this bias to determine the grade through discussion. On the other hand this same discussion could have lead to a milder evaluation. To be sure the automated process of labelling the strategy worked the results were checked by hand by two experts. We are aware of the fact we used pairs of students. The results might not represent individual students.
