New NER Results Sample Clauses

New NER Results. We have used these entities as new training data, using the same CRF model as mentioned in the introduction (Brandsen, 2018), and have seen a large increase in the overall micro F1 score, from 0.51 to 0.70, showing that this data is of better quality than the previously used training data. The difference between this, and the F1 between five human annotators (0.95) indicates that there is also still room for improvement. In Table 3.5 we show the difference in F1 score per entity type. Most types see a substantial increase, especially Locations, while the Material category sees Old New Difference Artefact 0.51 0.63 +0.12 Time Period 0.57 0.69 +0.12 Location 0.26 0.66 +0.40 Context 0.58 0.84 +0.26 Material 0.54 0.39 -0.15 Species n/a 0.49 n/a Overall Micro F1 0.51 0.70 +0.19 Table 3.5: F1 scores for entity types and overall micro F1 compared between the previous and new data set. Species wasn’t included in old data set, so we only present the score for the new data set. 0.75 F1 Entity type Artefact Context Location Material Overall Species Time Period 0.00 8 9 10 Number of chunks used as training data 3.1: CRF F1 score for each entity type per 1/10th chunk of data added to the training set. a decrease in F1 score. We wondered if this could be explained by the fact that we have much fewer annotations for the Material category, only 1,078 while all other categories have at least double that amount. To assess this, we divided the data set into 10 chunks, and retrained the CRF model 10 times, every time adding one more chunk of data. In Figure 3.1 we have plotted the F1 score for individual entity types and the overall micro F1 score for each model. Even though there are some fluctuations, it is evident that after adding a certain percentage of the data, the F1 scores for all the entity types plateau, even for the Material type. This probably indicates that the amount of annotations is sufficient and adding more data won’t substantially increase the F1 scores, although redundancy and noise in the data set could also potentially cause similar results. We will investigate this further in future research. The Species category performs similarly as Material, at 0.49, this could pos- sibly be explained by the fact that Species are written in both Dutch and Latin, but more work needs to be done to see if this is indeed the case. We also per- formed this analysis but instead of adding 10% of the data each time, we added 0.3 0.0 0.0 0.2 0.0 0.0 20.6 0.3 0.8 0.1 0.3 0.3 ...