CEROP
Stage 06 · Weeks 7–8

Explainability & the demo

Can someone who isn't a data scientist see why a case was flagged, and trust that the reason is real?

Export once, load read-only

The final models and their preprocessors were saved once from the notebooks. Everything after that loads them and never refits or re-saves them. The scoring and explanation notebooks print a fingerprint (SHA-256) of each file before and after running, so you can see nothing changed.

Chaining the saved artifacts end to end turned up two mismatches between the Week 4 operations preprocessor and the Week 6 model:

  • Column count. The preprocessor still emits the 8 Order Status columns that Week 6 dropped. Scoring reindexes to the model's own feature list, which removes exactly those columns.
  • One column's scale. The Week 6 training data capped and rescaled Order Item Profit Ratio, and the Week 4 preprocessor doesn't. On the eight demo orders it moves scores by under one percentage point and changes no decision (notebook 06). The site and console avoid the issue by using the model's own test file.

Real SHAP, checked case by case

Each explanation comes from SHAP values computed directly from the exported model files, in probability units, so a driver's number is in the same units as the score. For XGBoost that takes the interventional method with a background sample from the training data. For the Random Forest the tree leaves already hold probabilities, so the standard method works as is.

Before any case is shown, one check has to pass: the starting point plus every feature's contribution must equal the model's score to within one in a million. If one case fails, the export stops. All 24 financial and 40 operations cases in the console passed.

What moves bankruptcy scores most, on average

Mean absolute SHAP value across all 1,364 test firms, in percentage points

Source: results/web/financial_shap_cases.json (notebook 07).

Findings the checks surfaced

  • Scheduled days ranked first for shipments. The first real SHAP run put days for shipment (scheduled) at the top for every demo order, above shipping mode. The reason: in this data each shipping mode has exactly one scheduled window, so the two columns carry the same information and the model splits the credit. The console now says so next to every operations explanation.
  • One-hot columns split a driver in two. Shipping mode was encoded as several yes/no columns, and ranking those separately both understated it and produced labels like "Standard Class" on orders that weren't Standard Class. The fix adds those columns back into one driver and labels it with the order's real value.
  • A stale label. Cross-checking the demo against its source data found several orders showing the wrong shipping mode in their card subtitle. The explanation numbers were right; the caption above them wasn't. Fixed against the source rows.
  • Top drivers aren't the whole story. For some firms the top few factors explain well under half the move from the starting point. Every panel shows the remainder as its own line instead of hiding it.

The console

The original demo was a single HTML file built for a recorded class presentation. After the course I rebuilt it as the Risk Console: real held-out test cases, a threshold explorer over all 34,393 test orders, and a what-if panel that re-scores a firm in the browser. The browser version of the XGBoost model was checked against Python on all 1,364 test firms before it shipped.

Try the demo