What we did, layer by layer
Financial (Taiwan bankruptcy)
- Capped the lightly damaged ratios at their 99th percentile and dropped the columns where most values were placeholders.
- Built two composite features from related ratios, such as an average of three return-on-assets measures.
- Split 80/20 with stratification, so both sides keep the same 3.2% bankruptcy rate, then scaled using the training split only.
Operations (DataCo orders)
- Removed personal data (names, emails, passwords, street addresses), free text, and duplicate ID columns.
- Removed every field that's only known after delivery: actual shipping days, delivery status and the shipping date.
- Removed orders from 1 October 2017 onward, the unreliable tail found in Week 3. That took the data from 180,519 to 171,962 rows. This filter runs before the split because it's about data quality, not something learned from the data.
- Split the order date into year, month, day of week and hour, and joined the GSCPI value for each order's month.
- Clipped outliers in benefit per order with bounds learned from the training split, then encoded and scaled.
| Encoder | Columns | Output columns |
|---|---|---|
| StandardScaler | Days for shipment (scheduled), Benefit per order, Order Item Discount, Order Item Discount Rate, Order Item Profit Ratio, Order Item Quantity, Sales, Product Price, Order_Year, Order_Month, Order_DayOfWeek, Order_Hour, GSCPI | 13 |
| OneHotEncoder(drop="first") | Type, Customer Country, Customer Segment, Order Status, Shipping Mode | 17 |
| TargetEncoder | Category Name, Department Name, Market, Order Country, Order Region | 5 |
The rule that held everything together
Split first, then fit. Every step that learns something from data (capping thresholds, clipping bounds, scalers, one-hot vocabularies, target encodings) was fitted on the training split and only applied to the test split. If any of them had seen the test data, the test scores would have been quietly optimistic.
Checks the pipeline had to pass
The integration notebook stops with an error unless all of these hold, for both layers:
- no missing values in the processed training or test matrices;
- every value finite (no infinities left over from ratios);
- training and test matrices have the same columns.
These are the assert statements in notebook 03 (cells 10 and 32). An earlier, superseded pipeline run also left a checks file in the course folder; it isn't used here because it belongs to a different column set.
A leak we only caught later
Week 4 kept eight Order Status columns, reasoning that an order's status is known before delivery. In Week 6 the team looked again and judged they could carry information about how the delivery turned out, so they were dropped. The operations model went from 35 inputs to 27. The Week 5 operations results on the next page were measured before that fix, which is why the final operations numbers come from Week 6.
Per-layer cleaning: 02a financial · 02b operations