Nested Cross Validation Sample Clauses

Nested Cross Validation. The classifier performance is evaluated by means of nested cross validation (cv) and external dataset validation (Table 6). In the case of nested cv, the inner loop is responsible for model selection/hyperparameter tuning (validation set), while the outer loop is for error estimation (test set). A stratified four-fold resampling is used for the outer loop. Model parameters are tuned using grid search against AUC metric with 10-fold cross validation. For the results presented here, class imbalance was handled using smote subsampling, performed inside the inner cv resampling. Exclusion of well-being psychological variables (HADS depression and anxiety, C30 Emotional function, distress thermometer) is associated with poorer performance, however, the ability of the model to distinguish between the two classes is good in both cases (mean AUC>0.8) (Figures 18 & 19). Better tuning of the smote subsumbling is required to better balance sensitivity and specificity. Table 6: Performance of Random Forest Classifier. Average values are given. All data No HADS, C30 Emotional function, distress thermometer (during training) Accuracy 0.83 0.80 Balanced Accuracy 0.80 0.72 Precision 0.69 0.65 Sensitivity 0.71 0.56 Specificity 0.88 0.89 AUC 0.88 0.85 F1 0.70 0.60 Figure 18: ROC curves for each outer fold of the nested cross validation. Model with all the predictors. Figure 19: ROC curves for each outer fold of the nested cross validation. Model without HADS scales (depression, anxiety), C30 Emotional Function and distress thermometer.