CEROP
Stage 02 · Week 3

Data & exploration

What do the datasets actually contain, and what will bite us later if we don't deal with it now?

Three public datasets

Taiwan bankruptcy firms
6,819
DataCo order lines (raw)
180,519
GSCPI
monthly
  • Taiwanese Bankruptcy Prediction (UCI). Financial ratios for firms listed in Taiwan, 1999 to 2009, with a bankrupt yes/no label.
  • DataCo Smart Supply Chain (Mendeley Data). Order lines from a global retailer's shipments, with a late-delivery label.
  • Global Supply Chain Pressure Index (Federal Reserve Bank of New York). One number per month for how stretched global supply chains were.

Two very different balances

The first chart we made decided a lot. Bankruptcies are rare. Late shipments are common.

Counted from the label files in notebook 07 (results/web/dataset_facts.json).

What we found

Sentinel values pretending to be clean data

The bankruptcy data reported no missing values, which looked too good. Most ratios sit between 0 and 1, but some columns had maximums in the billions with minimums of zero. Those are placeholders for missing data, not real ratios. We sorted the affected columns by how much of each was damaged:

TierFeaturesPlan
Minor (< 2% affected)15Capped at 99th percentile
Borderline (~3% affected)1Capped, same treatment as minor tier
Severe (18–88% affected)8Dropped entirely
Source: Final Report Tables 1 and 2, extracted by scripts/extract_report_tables.py.

None of the badly damaged columns were among the strongest predictors, so dropping them cost little.

An early leakage warning

In the shipment data, the column most correlated with lateness was days for shipping (real): the number of days the shipment actually took. That's only known after delivery. A model trained on it would look excellent and be useless, because it would be reading the answer. It was flagged here and removed in Week 4, along with the delivery-status field. Days for shipment (scheduled) is known when the order is placed, so it stayed.

An unreliable tail

Plotting daily sales and profit over time showed a steady pattern until about September 2017, then sudden spikes and drops. That looks like a change in how data was collected, not a change in the business, so the tail was marked for removal.

What changed because of it

Week 4 got a concrete to-do list: tiered cleaning for the financial ratios, leakage columns out, outliers in profit fields clipped using training data only, and the post-September 2017 records removed.