The No. 1 Question to Ask When Evaluating AI Tools

Developers in the artificial intelligence (AI) sector often make claims about the accuracy of their AI tools to sell them to prospective customers. However, evaluating these tools can be challenging for non-experts, leading to disappointing or risky implementations. In an 11-month investigation, man...

Full description

Saved in:
Bibliographic Details
Published in:MIT Sloan management review 2023-04, Vol.64 (3), p.27-30
Main Authors: Lebovitz, Sarah, Lifshitz-Assaf, Hila, Levina, Natalia
Format: Article
Language:English
Subjects:
Artificial intelligence
Customers
Learning
Managers
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Developers in the artificial intelligence (AI) sector often make claims about the accuracy of their AI tools to sell them to prospective customers. However, evaluating these tools can be challenging for non-experts, leading to disappointing or risky implementations. In an 11-month investigation, managers in a healthcare organization conducted pilot studies of five AI tools and found that some performed poorly despite promising results. The key to assessing the quality of an AI tool lies in understanding its ground truth, which refers to the data used to train and validate the algorithm. Ground truth is the correct answer to the prediction problem that the tool is learning to solve. Managers should investigate the ground truth used by AI vendors and consider its quality and validity. They should also compare the AI ground truth to the ideal standard for experts in the specific domain. Overlooking shaky AI ground truth data can have severe consequences, so it is crucial for managers to thoroughly evaluate the ground truth before implementing AI tools.
ISSN:1532-9194