CEROP
Stage 04 · Week 5

Model selection

Which algorithm handles each layer best, judged by the metrics we committed to in Week 2?

Financial layer: three candidates

Each model was trained on the 5,455 training firms and scored once on the 1,364 test firms, at the default threshold of 0.50. Class imbalance was handled inside the models (class weights for Logistic Regression and Random Forest, a positive-class weight for XGBoost) rather than by inventing synthetic firms.

Recall, precision and F1 on the test set

Higher is better. Threshold 0.50.

Source: Final Report Table 7 (results/reported/final_report_table_07.csv).

Why XGBoost won

Logistic Regression catches the most bankruptcies, with recall of 0.818, but its precision is 0.186: most of the firms it flags are fine. An analyst would drown in false alarms. XGBoost has the best F1 (0.529) and the best ROC-AUC (0.950). Random Forest sits close behind on both, at 0.489 and 0.943, which makes it a reasonable fallback.

XGBoost on the test set

Said solvent
Said bankrupt
Really solvent
1,300correctly cleared
20false alarms
Really bankrupt
21missed
23caught

For a credit analyst, that reads as: of the 44 firms that went bankrupt, the model flagged 23. It flagged 43 firms in all, and 23 of those really failed. It's a strong pointer to which files to read first, and a weak tool for deciding on its own.

Official figures: results/week6_financial_tuning_results.csv and week6_financial_confusion_matrices.csv, row "XGBoost - Week 5 published", identical to the report. The Week 5 notebook in this repo was re-run later on different library versions and now shows F1 0.533 for XGBoost and 0.472 for Random Forest. Why the numbers moved.

Operations layer: three candidates

For late delivery the team compared Logistic Regression, a linear SVM and a Random Forest. Random Forest led on every metric that matters here. The linear models can't capture interactions such as shipping mode combined with order timing and region.

Week 5 comparison on the 35-feature data

Measured before the Order Status columns were removed in Week 6, so treat these as a ranking, not final numbers.

Source: results/week5_operations_model_comparison.csv.

What changed because of it

XGBoost went forward for suppliers and Random Forest for shipments. The question for Week 6 was whether tuning could improve either one.