Final numbers
| Layer | Model | Threshold | Precision | Recall | F1 | ROC-AUC | PR-AUC |
|---|---|---|---|---|---|---|---|
| Financial | XGBoost | 0.50 | 0.535 | 0.523 | 0.529 | 0.950 | 0.476 |
| Operations | Random Forest (tuned) | 0.50 | 0.845 | 0.629 | 0.721 | 0.826 | 0.873 |
| Operations | Random Forest (tuned) | 0.40 | 0.710 | 0.825 | 0.763 | 0.826 | 0.873 |
In plain terms: the supplier model puts a failing firm above a healthy one about 95% of the time and catches about half of bankruptcies at its default setting. The shipment model, once its threshold is set to 0.40, flags about 83% of late orders, and about 71% of its flags turn out to be right.
The Week 2 proposal set a recall target for the financial layer that this model doesn't reach at 0.50. The Week 6 search for a recall-first setting showed the price: pushing XGBoost's recall to 0.795 dropped precision to 0.220 (Final Report Table 11), so most alarms would be false. That trade-off is a decision for whoever uses the tool, which is why the console shows it rather than hard-coding one answer.
Reproducibility drift, as a lesson
Re-running the same notebooks on a different machine didn't give identical numbers:
- The exported XGBoost scores F1 0.533 locally against 0.529 in the report. Same code, same data, same random seed. The difference traces to library versions between Colab and the local environment. With 44 positives, a single changed prediction moves F1 noticeably.
- The Week 6 operations notebook picked a threshold of 0.39 on Colab and 0.40 on the local re-run, because the validation F1 values for neighbouring thresholds sit very close together.
Nothing was retrained to force a match. The official financial figure stays the reported one, the operations numbers follow the run that produced the shipped model, and both runs' files are kept. The lesson I took: pin library versions from the first week, and store results with the environment that produced them. This repo now has a pinned requirements.txt.
Limits
- Few failures to learn from. 44 bankrupt firms in the test set means wide uncertainty on every financial metric.
- About half of bankruptcies missed at 0.50. Recall 0.523. It's a screening aid, not an automated decision.
- One market, one decade. Taiwanese firms, 1999 to 2009. Not validated on other markets or recent years.
- A teaching dataset for shipments. DataCo is public research data, not a live company feed.
- Two drivers that are one. Scheduled days and shipping mode carry the same information in this data. Read them together.
- Mismatched saved artifacts. The saved operations preprocessor outputs 35 columns while the model expects 27, and it scales one column differently from the model's training data. Scoring has to work around both.
- Explanations show the top drivers. The rest are summed into one line, which is sometimes large.
- Tuning didn't help either model at the default threshold.
- Prototype only. No monitoring, retraining, access control or live data feed.
What I'd do next
- Put error bars on every headline metric. Week 6 bootstrapped the tuning comparison; doing the same for the final scores would show how wide "F1 0.529" really is.
- Validate on newer, non-Taiwanese filings, for example public company data from another market, before trusting the financial layer anywhere else.
- Fix the artifact chain: export one pipeline object per layer that goes from raw input to probability, so scoring can't drift from training.
- Choose thresholds from costs, not F1: ask what a missed delay and an unnecessary check actually cost, then pick the line that minimises the total.
- Check the GSCPI's real contribution with an ablation, since it was a design goal from Week 2.