The honest negative result
Both layers went through a randomized hyperparameter search with cross-validation. Neither tuned model beat its starting configuration at the default threshold of 0.50.
| Layer | Configuration | F1 at 0.50 |
|---|---|---|
| Financial | Week 5 settings, refit on the tuning split | 0.568 |
| Financial | Tuned XGBoost, same split | 0.505 |
| Operations | Baseline Random Forest | 0.724 |
| Operations | Tuned Random Forest | 0.721 |
That's worth saying plainly instead of burying. With 44 bankruptcies in the test set, the financial difference is a couple of firms either way. The Week 6 notebook checked this with a paired bootstrap (2,000 resamples of the test set): the 95% interval for the change in PR-AUC runs from -0.135 to 0.025, which includes zero, so the drop can't be told apart from noise. On the operations side the tuned forest did rank orders a little better (ROC-AUC 0.826 against 0.814), it just didn't turn that into a better F1 at 0.50. The financial layer therefore kept its Week 5 XGBoost.
The financial search fit on a 4,364-firm tuning split and held out 1,091 firms for validation (Final Report Table 9), so its F1 values aren't directly comparable with the full-training-set result of 0.529.
The threshold is a business decision
A classifier outputs a probability. Calling an order "likely late" at 0.50 is a convention, not a law. Lower the line and you catch more late orders but raise more false alarms. Raise it and the reverse happens. Which side is cheaper depends on the business.
For operations, the team picked the threshold with the best F1 on a validation set carved out of the training data, so the test set stayed untouched until the very end.
Validation precision, recall and F1 across thresholds
Drag the slider or click the chart. The bold line is the chosen threshold.
Source: results/week6_operations_threshold_results.csv. The best validation F1 was 0.748 at 0.40.
Before and after, on the test set
Tuned Random Forest at 0.50
Same model at 0.40
All three operations configurations
Source: results/week6_operations_tuning_results.csv and week6_operations_confusion_matrices.csv, from the Week 6 notebook as saved in this repo.
A note on the two runs
The Week 6 notebook was run twice. The team's Colab run, which the submitted reports quote, selected a threshold of 0.39 and reached recall 0.849 and F1 0.764. The notebook was re-run locally in August to export the final model, and that run selected 0.40. The model file behind the console comes from the second run, so this site uses its numbers everywhere. Both sets of result files are in the repo.
What changed because of it
The financial layer stays at 0.50, because tuning and threshold moves didn't give a dependable gain on 44 positives. The operations layer ships at 0.40, and the console lets you move that line yourself.