Three public datasets
- Taiwanese Bankruptcy Prediction (UCI). Financial ratios for firms listed in Taiwan, 1999 to 2009, with a bankrupt yes/no label.
- DataCo Smart Supply Chain (Mendeley Data). Order lines from a global retailer's shipments, with a late-delivery label.
- Global Supply Chain Pressure Index (Federal Reserve Bank of New York). One number per month for how stretched global supply chains were.
Two very different balances
The first chart we made decided a lot. Bankruptcies are rare. Late shipments are common.
Counted from the label files in notebook 07 (results/web/dataset_facts.json).
What we found
Sentinel values pretending to be clean data
The bankruptcy data reported no missing values, which looked too good. Most ratios sit between 0 and 1, but some columns had maximums in the billions with minimums of zero. Those are placeholders for missing data, not real ratios. We sorted the affected columns by how much of each was damaged:
| Tier | Features | Plan |
|---|---|---|
| Minor (< 2% affected) | 15 | Capped at 99th percentile |
| Borderline (~3% affected) | 1 | Capped, same treatment as minor tier |
| Severe (18–88% affected) | 8 | Dropped entirely |
None of the badly damaged columns were among the strongest predictors, so dropping them cost little.
An early leakage warning
In the shipment data, the column most correlated with lateness was days for shipping (real): the number of days the shipment actually took. That's only known after delivery. A model trained on it would look excellent and be useless, because it would be reading the answer. It was flagged here and removed in Week 4, along with the delivery-status field. Days for shipment (scheduled) is known when the order is placed, so it stayed.
An unreliable tail
Plotting daily sales and profit over time showed a steady pattern until about September 2017, then sudden spikes and drops. That looks like a change in how data was collected, not a change in the business, so the tail was marked for removal.
What changed because of it
Week 4 got a concrete to-do list: tiered cleaning for the financial ratios, leakage columns out, outliers in profit fields clipped using training data only, and the post-September 2017 records removed.