Why true success in machine studying goes past optimizing a single metric
“The brand new credit score mannequin didn’t carry out as anticipated,” he mentioned.
My colleagues from one other crew got here out from the “conflict room” trying dejected.
This was my first time witnessing a knowledge science “mistake” in a enterprise context, so I didn’t fairly perceive the size of it.
As I’d be taught over the following six months, this was a really pricey error from each a monetary and operational perspective, due to the downstream affect and reverberating results it had throughout the corporate.
This subject impacted each crew and all of us labored tirelessly for months after:
- Analysts labored on metrics to trace portfolio well being,
- Information scientists labored on diagnosing the difficulty and patching it,
- Engineers labored with information scientists to deploy the brand new mannequin,
- Product entrepreneurs labored on campaigns and messaging,
- Operational of us labored on buyer outreach and schooling on reimbursement.
The primary signal of bother was a lot greater default charges relative to expectations, however this was simply the…