Best Methods per Category. The above section shows which approach and parameter settings lead to the high- est average F1 scores, and here we investigate if we can achieve a higher average F1 score by combining the best approaches and settings for each individual cat- egory. The results for time periods and site types are shown in Tables 4.7 and 4.8, respectively. For time periods, combining the best method per individual category leads to an average F1 score of 0.710, which is a slight decrease compared to the 0.719 of the settings with the best F1 average over all categories. This again can be explained by the quality of the development sets: by using the optimal param- eter settings for a category obtained on the development set, it unfortunately does not imply that these settings are (close to) optimal on the test set. This phenomenon is similar to that observed in the previous section, were the best parameter settings for the test set ranked sixth on the development set. For the site types, the opposite shows, as we find an average increase of 0.133 compared to the highest scoring settings on the development set. Moreover, the F1 score of 0.542 – the result of optimal settings for the test set – is met. It has to be noted that we find F1 scores of 0.0. These categories are barely represented within our test set, and for these it is difficult to determine the quality of the classification process: a recall of 0.0 is frequent. The MultNB classifier does not appear in the top ten. We expected to see that balancing the training set would have a positive effect on the classification process for this classifier, but this is not reflected by our results. However, it is interesting to see that balancing the training set has a positive effect on the classification process of SVM for neo and ijz, despite the theoretical unbalanced data set ‘protection’. Again, this can be explained by the random influence on 68 CHAPTER 4. TEXT CLASSIFICATION Category PP Aug Bal List Precision Recall F1 score meso 134 2 No Gen 1.0 0.550 0.710 paleo 123 2 No Gen 1.0 0.385 0.555 brons 158 2 No Gen 0.714 0.435 0.541 neo 123 2 Yes Gen 0.653 0.630 0.642 rom 128 0 No Gen 0.952 0.741 0.833 ijz 134 0 Yes Gen 0.828 0.828 0.828 nt 134 0 No Gen 0.722 0.848 0.780 xme 1236 0 No Gen 0.764 0.823 0.792 average - - - - 0.829 0.655 0.710 Table 4.7: Overview of the best methods per individual category for time period classification and the overall average of these best methods. Column names yield the meaning as provided in ...
Best Methods per Category. The above section shows which approach and parameter settings lead to the high- est average F1 scores, and here we investigate if we can achieve a higher average F1 score by combining the best approaches and settings for each individual cat- egory. The results for time periods and site types are shown in Tables 4.7 and 4.8, respectively. For time periods, combining the best method per individual category leads to an average F1 score of 0.710, which is a slight decrease compared to the 0.719 of the settings with the best F1 average over all categories. This again can be explained by the quality of the development sets: by using the optimal param- eter settings for a category obtained on the development set, it unfortunately does not imply that these settings are (close to) optimal on the test set. This phenomenon is similar to that observed in the previous section, were the best parameter settings for the test set ranked sixth on the development set. For the site types, the opposite shows, as we find an average increase of 0.133 compared to the highest scoring settings on the development set. Moreover, the F1 score of
