For an ML project to succeed within a business organization, it’s crucial to tie the performance of an ML system to the overall business performance
If an ML system is built for a business, it must be motivated by business objectives, which need to be translated into ML objectives to guide the development of ML models
- Data scientists care so much about ML metrics (improving a model’s accuracy from 94% to 94.2%) but in reality business managers don’t care unless it impacts business metrics either directly (increased sales) or indirectly (customers spening more time on company website)
- Many companies create their own metrics to map business metrics to ML metrics.
- Netflix measures the performance of their recommender system using take-rate: the number of quality plays divided by the number of recommendations a user sees.
- The higher the take-rate, the better the recommender system.
- It’s hard to see the direct effects of ML projects
- So experiments are often conducted
- Businesses choose ML models with better business metrics, not ML metrics
- Companites that payoff with ML often are very mature already
- How long it takes for a company to bring a model to production is proportional to how long it has used ML. (the longer it used, the shorter time in production)